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ABSTRACT 


This  thesis  provides  a  framework  to  model  human  belief  and  misperception  in  helicopter 
overland  navigation.  Helicopter  overland  navigation  is  a  challenging  mission  area 
because  it  is  a  complex  cognitive  task,  and  failing  to  recognize  when  the  aircraft  is  off- 
course  can  lead  to  operational  failures  and  mishaps.  A  human-in-the-loop  experiment  to 
investigate  pilot  misperception  during  simulated  overland  navigation  by  analyzing  actual 
navigation  trajectory,  pilots’  perceived  location,  and  corresponding  confidence  levels  was 
designed.  Fifteen  military  officers  with  prior  overland  navigation  experience  completed 
four  simulated  low-level  navigation  routes,  two  which  entailed  autonavigation.  Analysis 
shows  that  there  is  not  a  negative  correlation  between  perceived  and  actual  location  of  the 
aircraft,  inferring  that  confidence  is  not  a  good  indicator  of  performance.  There  is 
however  some  evidence  that  there  is  a  negative  correlation  between  perceived  location 
and  intended  route  of  flight,  suggesting  that  there  is  a  bias  towards  that  intended  flight 
route.  If  aviation  personnel  can  proactively  identify  the  circumstances  in  which  usual 
misperception  occur  in  navigation,  they  may  reduce  mission  failure  and  mishap  rate. 
Fleet  squadrons  and  instructional  commands  can  benefit  from  this  study  to  improve 
operations  that  require  low  level  flight  while  also  improving  crew  resource  management. 
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EXECUTIVE  SUMMARY 


Objective: 

This  thesis  provides  a  framework  to  model  human  belief  and 
misperception  in  helicopter  overland  navigation.  Helicopter  overland  navigation  is  a 
challenging  mission  area  because  it  is  a  complex  cognitive  task,  and  failing  to  recognize 
when  the  aircraft  is  off-course  can  lead  to  operational  failures  and  mishaps. 

Methods: 

Systematic  biased  perception  during  an  overland  navigation  was  observed 
in  Sullivan  et  al.  (2010).  In  the  current  study,  we  design  a  human-in-the-loop  experiment 
to  investigate  pilot  misperception  during  simulated  overland  navigation  by  analyzing 
actual  navigation  trajectory,  pilots’  perceived  location,  and  corresponding  confidence 
levels.  Fifteen  military  officers  with  prior  overland  navigation  experience  completed  four 
simulated  low-level  navigation  routes,  two  which  entailed  autonavigation. 

Results: 

Data  was  collected  regarding  the  amount  of  time  the  participants  spend  in 
the  “dangerous”  off  track  and  high  confidence  area  of  perception,  the  correlation  between 
perception  and  confidence,  and  the  relation  of  confidence  and  error  versus  time.  Data  was 
categorized  into  four  quadrants  based  off  perception  error  and  corresponding  confidence 
levels;  eg,  “On-track”  with  “High”  or  “Low”  confidence  and  “Off-track”  with  “High”  or 
“Low”  confidence.  Subjects  were  “On-track”  and  had  a  corresponding  “High” 
confidence  58.37%  of  the  time,  but  the  second  most  frequent  state  was  the  dangerous 
quadrant,  “Off-track”  yet  still  confident  that  they  are  “On-track.”  Of  the  time  pilots  were 
“Off-track”  (34.65%),  they  had  wrong  perception  77.86%  of  the  time.  This  observation 
was  more  explicit  in  autonavigation  scenarios  at  81.55%.  Hypothesis  testing  was 
conducted  to  determine  if  there  is  a  negative  correlation  between  the  distance  between 
actual  and  perceived  location  of  the  aircraft  versus  confidence,  if  there  is  a  negative 
correlation  between  the  distance  between  the  perceived  location  of  the  aircraft  and 
intended  route  of  flight,  and  if  confidence  and  perception  error  increases  with  time  of 
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flight.  Analysis  shows  that  there  is  not  a  negative  correlation  between  perceived  and 
actual  location  of  the  aircraft,  inferring  that  confidence  is  not  a  good  indicator  of 
performance.  There  is  however  some  evidence  that  there  is  a  negative  correlation 
between  perceived  location  and  intended  route  of  flight,  suggesting  that  there  is  a  bias 
towards  that  intended  flight  route.  Lastly,  confidence  tends  to  decrease  while  perception 
error  increases  the  longer  the  pilot  navigates. 

Implication: 

If  aviation  personnel  can  proactively  identify  the  circumstances  in  which 
usual  misperception  occur  in  navigation,  they  may  reduce  mission  failure  and  mishap 
rate.  Fleet  squadrons  and  instructional  commands  can  benefit  from  this  study,  especially 
for  use  in  search  and  rescue,  anti-surface  warfare,  combat  search  and  rescue,  and  naval 
special  warfare  operations  because  of  the  low-level  navigation  flight  profiles  required. 
This  study  can  also  improve  crew  resource  management  inside  the  helicopter  cockpit. 
Helicopter  crews  are  heavily  reliant  on  each  member  of  the  crew,  and  additional 
complacency  can  occur  when  one  of  the  members  is  confident  that  they  are  on  course. 


KEYWORDS:  Training,  Simulation,  Human  Factors,  Aviation,  Helicopter,  Navigation, 
Confidence,  Overconfidence,  Perception,  Misperception,  Bayes 
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I.  INTRODUCTION 


A.  RESEARCH  OBJECTIVE(S) 

The  goal  of  this  research  is  to  understand  pilot’s  perception  and  confidence  during 
overland  navigation,  and  why  these  may  lead  to  visual  misperception.  Mishaps  and 
mission  failure  have  been  linked  to  improper  navigation,  and  these  problems  increase 
when  the  pilot  is  unaware  of  their  miscalculations.  Although  it  is  recognized  that 
misperception  during  navigation  is  dangerous,  we  are  unaware  of  studies  that  use 
quantitative  analysis  to  link  the  specific  factors  of  why  pilot  misperception  is  prevalent. 
We  seek  to  address  this  critical  question  in  order  to  develop  training  and  tools  to  improve 
the  problem. 

In  this  study,  participants  were  placed  in  a  high  cognitive  workload  simulated 
environment  based  on  real-word  scenarios  and  their  performance  was  evaluated.  We 
evaluate  performance  by  measuring  the  distance  between  the  participant’s  perceived 
location,  compared  to  their  actual  location,  and  their  corresponding  confidence  level. 
Results  from  the  study  are  expected  to  be  used  to  improve  pilot  training,  crew 
coordination,  and  aircraft  or  cockpit  design  and  technologies. 

B.  BACKGROUND 

1.  Helicopter  overland  navigation 

Helicopter  overland  navigation  is  a  challenging  and  complex  cognitive  task. 
Helicopter  overland  navigation  is  comprised  of  a  number  of  sub-skills  that  require 
continuous  visual  cue  perception  and  decision  making.  The  “Flight  Training  Instruction, 
Instrument  and  Navigation  Advanced  Phase,  TH-57”  states  that  the  “Pilot  Not  At  the 
Controls  (PNAC)  is  primarily  responsible  for  accurate  navigation.  He  must  remain 
oriented  at  all  times,  monitor  cockpit  instruments,  and  perform  assigned  cockpit  duties  as 
briefed.  During  an  aircraft  or  system  emergency,  he  executes  the  emergency  procedures 
as  briefed  by  the  pilot”(CNATRA  P-458).  The  navigational  task  can  be  done  by  visual 
navigation,  dead  reckoning,  or  electronic  navigation  using  Global  Positioning  System 
(GPS),  Doppler  Radar,  or  some  other  system.  Visual  navigation  is  performed  by 
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comparing  terrain  features  on  a  map  to  what  is  seen  out  the  window  of  the  aircraft.  Dead 
reckoning  involves  taking  a  known  position  and  by  using  direction  and  timing.  Advances 
in  GPS  technology  have  decreased  the  use  of  visual  navigation  and  dead  reckoning,  yet 
can  still  only  be  used  as  a  form  of  backup  navigation  in  all  fleet  aircraft  and  a  critical  skill 
if  ever  required  to.  The  PNAC  may  use  all  three  forms  of  navigation,  separately  or  in 
combination,  in  order  to  accurately  navigate  through  low-level  terrain.  On  top  of  this 
heavy  navigational  workload,  the  PNAC’s  most  important  role  is  to  assist  the  Pilot  At  the 
Controls  (PAC)  in  obstacle  avoidance.  Because  of  the  complex  cognitive  task  placed  on 
the  nonflying  aviator,  it  is  easy  to  deviate  from  course.  Straying  off  course  is  not  an  issue 
if  the  aviator  is  aware  of  being  off  course.  However,  often  the  aviator  is  unaware  of 
being  off  course. 

2.  Misperception  and  Overconfidence 

Misperception  can  lead  to  both  mission  failure  by  the  aircraft  not  reaching  its 
intended  destination  on  time,  and  also  mishaps  due  to  the  pilot  flying  into  obstacles  in  the 
terrain.  The  Navy  Safety  Center  has  adopted  James  Reason’s  Swiss  cheese  model  for 
understanding  the  underlying  process  that  results  in  mishaps  (Reason,  2000).  The  Swiss 
cheese  model  relates  a  system  to  a  stack  of  slices  of  Swiss  cheese.  Each  slice  of  cheese  is 
a  layer  of  the  system,  and  the  holes  are  analogous  to  opportunities  for  the  system  to  fail. 
Mishaps  occur  only  when  the  holes  line  up  allowing  failures  to  pass  through  without 
being  stopped  by  another  system.  This  research  focuses  on  the  slice  that  relates  to  pilot 
judgment.  There  are  two  particular  recent  mishaps  that  expose  the  importance  of 
exploring  information  on  the  subject.  The  first  mishap  involved  an  MH-60S  with  17 
individuals  on  board.  In  this  mishap,  they  were  flying  in  a  new  area  when  unexpected 
bad  weather  arrived.  They  tried  to  deviate  from  the  intended  course  to  a  nearby  landing 
field.  The  crew  relied  heavily  on  visual  navigation  because  they  did  not  have  the  divert 
airfield  in  their  cockpit  navigational  computer.  This  lack  of  information,  along  with  the 
low  cloud  layer,  caused  the  crew  to  misperceive  their  location,  and  ultimately  lead  to  a 
crash  landing  in  the  snow  covered  West  Virginia  mountains.  The  second  mishap 
involved  a  senior  pilot  and  aircrew  flying  a  MH-60S  under  daytime  clear  atmospheric 

conditions.  The  senior  pilot  had  a  vast  knowledge  of  the  area,  but  decided  to  not  follow 
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course  rules  back  to  their  home  station.  They  hit  power  lines  and  crashed  the  helicopter. 
This  mishap  was  due  to  overconfidence  in  the  pilot  on  where  they  perceived  they  were 
flying.  The  pilot’s  confidence  contributed  to  the  complacency  of  the  co-pilot  and 
aircrew.  Luckily,  in  both  of  these  mishaps  no  one  was  killed,  but  it  did  highlight  the  fact 
that  misperception  can  have  drastic  consequences. 

3.  Related  research 

Table  1  presents  a  matrix  derived  from  signal  detection  theory  (SDT)  showing  the 
four  different  awareness  states  of  a  navigating  pilot  (Sullivan,  2010).  The  most 
concerning  area  is  marked  “Dangerous,”  where  the  crew  believes  that  they  are  on  course 
when  they  are  not.  This  thesis  focused  on  this  type  of  misperception  because  it  lends 
itself  to  mishaps  and  mission  failures. 


Assessing  Navigation 
Performance 

Confidence 

Low 

High 

Correctness 

Low 

Struggling.  No  accurate 
fix,  aware  that  aircraft  is 
off  track 

Dangerous.  Lost  and 
doesn’t  realize  it.  Positively 
misidentified  correlating 
features. 

High 

On  course  and  lucky. 
Accurate  fix,  but  not 
confident  in  navigation 
solution. 

Skilled  performer.  On  track 
and  certain. 

Table  1.  Matrix  for  Assessing  Navigational  Skills  (After  Sullivan,  2010).  We  see 
that  the  most  dangerous  combination  is  low  correctness  and  high 
confidence,  and  the  pilots  do  not  recognize  the  loss  of  situational  awareness. 


Sullivan  et  al.  (2010)  collected  navigation  and  eye  scan  pattern  data  from  12 
military  officers  who  underwent  an  overland  navigation  simulation.  This  experiment  had 
pilots  navigate  through  12  waypoints  in  a  simulation  terrain  model  of  Twentynine  Palms, 
CA.  Regression  analysis  confirmed  previous  results  that  flight  performance  measures 
such  as  RMS  (root-mean  square)  error  were  not  predicted  by  the  expertise  level  of  pilots 
(Bellenkes,  Wickens  &  Kramer  1997).  However,  Sullivan  et  al.  (2010)  found  that  gaze 
parameters  and  scan  management  skills  were  predicted  by  the  expertise  level.  Most 
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relevant  to  this  study,  analysis  through  Flight  and  Eye  Scan  visualization  Tool  (FEST, 
Figurel)  showed  that  some  pilots  had  biased  perception.  As  shown  in  Figure  1,  subject 
five  missed  a  waypoint  and  started  to  track  north  of  the  intended  route.  Instead  of  using 
available  visual  cues  on  the  flight  simulation  screen  or  out-the-window  (OTW)  to  realize 
that  he  was  off  course,  he  perceived  that  he  was  still  on  course.  This  pattern  suggests  the 
pilot  was  using  some  biased  visual  cueing  in  which  he  overweighed  OTW  cues  that  fit 
into  his  perception  that  he  was  on  course,  and  disregarded  OTW  cues  that  did  not  fit  with 
his  hypothesis. 


Figure  1.  Feft:  Flight  and  Eye  Scan  visualization  Tool  (FEST)  and  Right:  Subject 
five’s  actual  flight  trajectory  (blue)  and  planned  route  (black)  (right) 
(Sullivan  et  al.  2011  and  Yang  et  al.  2011) 


Yang  used  data  gathered  from  Sullivan’s  experiment  and  classified  pilot 
misperception  into  three  types  (Yang,  Kennedy,  Sullivan  &  Day,  2011).  Type  one  is 
confusion  between  inference  and  evidence.  This  type  is  especially  seen  when  the  pilots 
have  a  high  belief  that  they  are  on-track.  Type  two  is  when  pilots  incorrectly  assume 
mutually  exclusive  evidences  based  on  a  highly-likely  visual  cue  for  multiple  locations. 
The  third  type  is  when  pilots  discount  cues  that  do  not  correspond  to  their  hypothesis. 
This  misperception  type  can  be  attributed  to  inattentional  blindness  (Simons  &  Chabris, 
1999).  Table  2  summarizes  the  three  misperception  types  compared  to  Bayesian 
perception. 
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Perception  type 

Posterior  probability 

Bayesian  agent  (A) 

...  p(d\H)-p(H ) 

P(  1  J  p(d\H)-p(H)  +  p(d\~H)-p(~H) 

Misperception  Type  1  (Bl) 

p(H\d )  =  p(d\H)  when  p(d\H)  ~  1 

Misperception  Type  2  (B2) 

v(Il\d)  ~  pMI »>p(»)  nOllcO  “  pMIh)p(H) 

when  p(d\H)  ~  1 

Misperception  Type  3  (B3) 

p(H\d)  =  p(H )  when  p(d\H)  ~  0 

Table  2.  Visual  misperception  modeling  using  a  Bayesian  framework  (From  Yang  et 
al.,  201 1);  where  d  =  terrain  features  that  the  pilot  sees,  and  H  =  pilots 

current  position 


Bayesian  updating  centers  on  the  fact  that  subjective  beliefs  should  be  updated 
with  the  addition  of  some  evidence.  Orban  claimed  in  his  paper,  “Bayesian  Learning  of 
Visual  Chunks  by  Human  Observers,”  that  humans  act  and  learn  as  logical  Bayesian 
agents  even  if  they  are  unaware  of  this  fact.  Humans  are  able  to  update  conditional 
probabilities  to  make  correct  maximizing  or  minimizing  choices  in  complex 
environments  (Orban,  Fiser,  Aslin  &  Lengyel,  2008).  For  this  type  of  Bayesian  cognitive 
modeling,  human  errors  are  not  considered.  These  errors,  to  include  misperception,  are 
very  important  in  determining  why  aviators  get  off  track.  Knowing  that  a  pilot  is  acting 
as  Bayesian  agent  is  not  as  useful  as  knowing  why  the  pilot  is  not  acting  as  a  Bayesian 
agent.  When  pilots  are  not  acting  as  a  Bayesian  agent  there  is  greater  chance  of  them 
getting  off-course. 

Inattentional  blindness,  or  perceptual  blindness,  occurs  when  a  person  is 
overwhelmed  with  inputs  causing  them  to  miss  a  stimulus  that  is  in  plain  sight.  Simons 
and  Chabris  (1999)  demonstrated  this  phenomenon  by  showing  a  video  of  two  groups  of 
people  passing  a  basketball  to  participants.  The  participants  were  supposed  to  count  the 
number  of  passes  made  in  the  video.  During  this  video  a  person  dressed  as  a  gorilla 
walks  through  the  scene.  Fifty  percent  of  the  subjects  tested  did  not  notice  the  gorilla  in 
the  video.  Sullivan  (2010)’s  data  showed  the  inattentional  blindness.  The  high  cognitive 
workload  experienced  in  helicopter  low  level  navigation  results  in  pilots  focusing  on 
certain  inputs,  for  example  gauges  map,  and  terrain.  However,  they  miss  other  inputs  that 
an  outsider  could  consider  obvious.  After  the  fact,  it  is  easy  to  show  pilots  that  they 
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missed  substantial  terrain  features,  just  as  it  was  easy  to  show  participants  that  they 
missed  the  gorilla  in  the  video.  OTW  gaze  data  showed  the  pilot  “looked”  at  the  terrain 
which  is  a  cue  for  realizing  that  they  are  off-track.  However,  they  did  not  “perceive”  the 
cue  they  were  looking  at.  During  pilot  training,  students  are  instructed  to  refrain  from 
fixating  on  one  input  and  to  keep  their  scan  moving,  yet  this  inattentional  blindness  is  still 
observed. 

Another  theory  on  why  pilot  misperception  occurs  is  overestimation  of  personal 
ability  or  overconfidence.  Overconfidence  leads  pilots  to  perceive  that  they  are  on  course 
even  after  they  have  drifted  off  course.  After  drifting  off  course,  pilots  will  try  to  match 
their  outside  surrounding  to  the  map,  and  not  the  converse.  This  overconfidence  can  be 
explained  by  Stone’s  research  of  self-efficacy  (Stone,  1994).  In  this  experiment,  he 
placed  subjects  in  complex  cognitive  tasks  and  observed  their  behavior,  performance,  and 
perceptions.  He  found  that  the  subjects  were  biased  towards  overestimates  of  personal 
ability;  that  is,  they  perceived  their  performance  to  be  better  than  it  actually  was.  This 
finding  seems  counterintuitive,  because  one  would  believe  that  they  would  be  less 
confident  when  they  are  engaged  in  high  workload  environment.  There  can  be  two 
reasons  for  this  overconfidence  applied  to  navigation.  The  first  is  that  the  aviator  is  so 
task  overloaded  that  they  rely  too  heavily  on  their  navigation  abilities  and  training.  Even 
if  the  pilot  believes  that  they  are  getting  behind  on  their  navigation,  they  still  believe  that 
they  are  heading  in  the  right  direction  because  of  previous  good  decisions  and  they  do  not 
have  adequate  time  to  get  their  precise  location.  The  second  reason  for  overconfidence  is 
due  to  a  lack  of  knowledge  of  the  current  situation.  If  a  pilot  has  limited  experience 
operating  in  a  certain  environment,  they  can  be  unaware  of  the  dangers  associated  with  it. 
An  aviator  who  has  never  flown  in  the  mountainous  desert  environment  may  apply 
incorrect  navigational  techniques  assuming  it  will  yield  the  same  results. 

C.  RESEARCH  QUESTIONS 

My  thesis  models  the  different  types  of  visual  misperception  and  validates  these 
classifications  by  making  comparisons  between  what  pilots  see  OTW  and  their  alleged 
location  on  the  map  to  the  actual  flight  trajectory  and  location  through  human-in-the-loop 

experiments.  The  thesis  focused  on  three  areas:  (1)  Investigate  correlation  between 
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pilot’s  perception  and  confidence.  (2)  Provide  misperception  modeling  in  a  Bayesian 
framework.  (3)  Given  what  we  learn  about  pilot  misconception,  what  training, 
operational  or  acquisition  strategies  should  we  recommend  to  the  Navy  to  increase 
combat  efficiency  and  decrease  risks? 


D.  OUTLINE  OF  THESIS 

Methods:  This  section  covers  the  reason  for  experimental  design  and 

experimental  hypotheses.  This  section  also  described  the  dependent  and  independent 
variables  used  and  how  they  were  collected.  Experimental  setup  and  procedures  also  fall 
under  this  section. 

Analysis  and  Results:  This  section  includes  how  experimental  data  was  filtered, 
grouped,  and/or  correlated.  The  outputs  of  this  analysis  are  presented. 

Model:  The  model  section  includes  model  design,  implementation,  and  output. 

Conclusions  and  Recommendations:  This  section  highlights  and  summarizes 
results  from  analysis  and  the  model.  This  section  also  provides  recommendations  based 
off  findings  and  contributions  of  the  study.  Future  work  suggestions  are  then  presented. 
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II.  SIMULATOR-BASED  HUMAN -IN-THE-LOOP 

EXPERIMENTS 

A.  EXPERIMENTAL  HYPOTHESES 

This  section  provides  the  reasoning  for  the  experimental  setup  used  to  test  initial 
hypotheses.  The  goal  of  the  experiment  was  to  place  a  participant  with  navigation 
experience  in  a  situation  where  only  visual  terrain  cues  were  available  for  navigating. 
Four  hypotheses  were  constructed  that  focused  on  participants’  perceived  and  actual 
location  and  self-assessed  Figure  of  Merit,  or  confidence,  during  the  visual  navigation 
tasks.  These  four  hypotheses  will  be  analyzed  in  the  following  chapters. 

The  first  null  hypothesis  is  there  no  correlation  between  distance  from  actual 
helicopter  position  during  pause  points  and  the  participant’s  perceived  location  and  the 
participant’s  confidence  of  their  perceived  location.  This  hypothesis  is  based  on  the  idea 
that  one’s  confidence  is  not  related  to  their  navigation  performance,  or  that  one’s 
confidence  may  even  increase  with  greater  navigation  error.  The  alternative  hypothesis 
was  that  there  is  a  negative  correlation  between  confidence  and  perception  error.  This 
hypothesis  states  that  if  a  participant  perceived  themselves  to  be  “lost,”  they  would  be 
able  to  recognize  that  fact  and  therefore  have  a  corresponding  low  confidence  level.  This 
hypothesis  test  can  also  be  related  to  unrecognized  special  disorientation.  Unrecognized 
disorientation  is  considered  the  most  dangerous  of  the  three  types  of  disorientation,  and  is 
when  pilots  are  unable  to  correctly  perceive  what  is  happening  in  their  surroundings. 
Failing  to  reject  the  null  hypothesis  for  this  case  could  be  a  causal  factor  for  pilots  getting 
off-track,  along  with  the  associated  mishaps  and  mission  failures. 

The  second  null  hypothesis  tested  is  no  correlation  between  the  pilot’s  confidence 
and  the  distance  between  their  perceived  location  and  the  intended  route  of  flight.  This 
hypothesis  claims  that  when  a  pilot  believes  that  they  are  off  the  intended  route  of  flight, 
there  would  not  be  a  corresponding  low  confidence  level.  The  alternate  hypothesis  is  that 
there  is  a  negative  correlation  between  confidence  and  distance  between  perceived  and 
actual  course.  The  reasoning  behind  this  alternative  hypothesis  was  because  if  a  pilot 

was  maintaining  an  accurate  track  and  course,  they  would  be  close  to  the  intended  route. 
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Once  a  pilot  believed  that  they  have  strayed  off  course,  they  would  be  more  likely  to  be 
guessing  at  their  current  position.  Failing  to  reject  the  null  could  show  that  pilots  have  a 
biased  confidence  belief  even  when  they  are  not  tracking  on  course. 

The  longer  a  participant  navigates  through  an  intended  route,  the  greater  the 
distance  between  the  perceived  location  and  the  actual  helicopter  position  was  the  third 
null  hypothesis  tested.  The  reasoning  behind  this  null  hypothesis  was  because  it  would 
be  assumed  that  the  longer  a  pilot  navigates  through  a  scenario  the  more  likely  they 
would  deviate  from  their  intended  course.  The  causal  factors  for  this  could  be  because 
the  participant  has  more  time  to  stray  off  course,  fatigue,  or  leg  heading  or  timing  was 
forgotten  or  not  as  fresh  in  the  mind  compared  to  earlier  in  the  navigation  route.  The 
alternative  hypothesis  was  that  there  not  an  increase  in  distance  between  perceived  and 
actual  location  the  longer  into  the  participant  navigates.  Rejecting  the  null  hypothesis 
would  state  that  pilot  misperception  is  not  reliant  on  where  on  the  route  the  aircraft  is 
located. 

The  final  hypothesis  that  was  tested  was  that  confidence  decreased  the  further  into 
the  navigation  route,  or  confidence  is  a  function  of  time.  The  null  hypothesis  assumes 
that  the  confidence  level  of  participant  would  reduce  because  of  some  causal  factors. 
These  factors  could  be  because  the  pilot  had  more  time  to  realize  that  they  were  off 
course,  fatigue,  or  overtasked.  The  alternative  to  the  null  is  that  confidence  does  not 
decrease  the  longer  the  participant  was  flying.  Results  regarding  this  hypothesis  would 
give  insights  into  how  pilots  generate  their  confidence  levels  over  a  period  of  time. 

B.  EXPERIMENTAL  METHOD 

1.  Task  Definition 

Low  level  navigation,  or  “terrain  flight”  (TERF),  is  defined  as  flight  below  200 
feet  above  ground  level  (AGL)  (CNATRA  P-458).  This  environment  is  challenging 
because  the  low  flight  level  reduces  the  amount  of  terrain  that  the  pilot  can  see,  and 
requires  intense  emphasis  on  flight  parameters.  We  focus  on  the  navigation  aspect  of 
terrain  flight;  our  simulated  aircraft  is  held  at  constant  altitude  and  does  not  experience 
emergencies  -  two  critical  dimensions  of  real-world  TERF  navigation.  “Proficient 
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navigation  during  low-level  flight  requires  training  and  practice.  Identifying  [check 
points]  (CPs)  is  the  critical  task,  since  this  requires  the  navigator  to  be  proficient  in  map 
reading,  terrain  interpretation,  and  correlation  of  terrain  features  with  map  symbols.  He 
must  be  able  to  visualize  from  the  map  how  the  terrain  along  the  flight  path  should  look. 
He  must  also  be  able  to  look  at  the  terrain,  identify  his  location,  and  locate  it  on  the  map” 
(CNATRA  P-458). 

Low-level  VFR  Navigation  requires  efficient  visual  scanning.  Visual  scanning  is 
the  ability  to  recognize  and  reference  key  terrain  features  in  a  given  field  of  view.  These 
key  terrain  features  will  allow  the  navigator  to  recognize  waypoints  and  intermediate 
check  points  along  the  route.  Pilots  must  also  “be  prepared  for  the  terrain  to  look 
differently  than  as  planned  and  adjust  as  necessary”  (CNATRA  P-458). 

2.  Navigation  Route  Design 

The  route  environment  and  waypoint  selection  plays  a  large  role  in  the  outcome 
of  this  study.  The  route  needed  to  be  in  a  location  that  did  not  favor  any  particular  pilot’s 
previous  Fleet  experience  and  covered  an  area  that  had  challenging  terrain  so  that  there 
was  great  possibility  of  misperceiving  the  surroundings.  Finally,  it  needed  to  be  an  area 
adequately  mapped  in  FALCONVIEW  to  use  in  our  analysis.  The  mountainous  area  of 
Twentynine  Palms  was  selected  for  this  experiment  for  several  reasons.  The  first  being 
that  the  area  includes  some  landmarks,  and  there  are  multitudes  of  executable  routes. 
Secondly,  most  of  the  participants  of  the  study  have  not  operated  in  this  area.  Finally,  we 
consider  the  high  altitude  desert  terrain  to  be  comparable  to  the  current  operating 
environments  in  Iraq  and  Afghanistan. 

After  choosing  the  operating  area,  routes  were  selected  to  support  hypothesis 
testing.  To  collect  sufficient  data,  four  routes  were  generated,  along  with  a  practice  route. 
The  participant  had  to  navigate  through  the  route  using  a  joystick  which  controlled 
heading  (roll)  only;  participants  had  no  control  over  pitch,  yaw,  power,  or  airspeed.  The 
pilots  did  not  have  to  control  attitude,  airspeed,  rotor  speed  or  ball,  therefore  greatly 
simplifying  the  navigational  task.  For  the  last  two  routes,  a  scripted  “autopilot”  guided 
the  participant  along  a  set  course  without  their  control. 
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The  autonavigation  routes  were  added  to  normalize  the  experiment  in  the 
following  manner:  If  all  of  the  participants  were  able  to  control  the  helicopter  through 
the  routes,  each  pilot  would  see  different  terrain  features  because  the  probability  of  two 
independent  pilots  flying  the  same  course  is  practically  zero.  Controlling  the  route  with 
the  autopilot  allowed  the  experiment  proctor  to  pause  the  route  at  the  exact  same  points, 
so  that  each  participant  sees  the  same  terrain.  More  than  this,  it  ensures  that  each  pilot  is 
presented  the  terrain  identically  -  that  they  approach  it  from  the  same  azimuth  and  roll 
angle. 

With  the  number  and  types  of  routes  chosen,  the  waypoints  comprising  the  route 
were  selected.  The  routes  needed  to  be  fair,  yet  challenging  enough  for  the  pilots  to  get 
off-course.  We  subjectively  created  routes  that  were  appropriate  for  a  late-first  tour 
aviator’s  level  of  experience. 

The  practice  route  was  designed  to  get  the  participant  familiar  with  the  control  of 
the  helicopter,  feel  comfortable  using  the  confidence  program,  establish  a  scan  pattern, 
and  gain  familiarity  with  the  interfaces.  The  practice  route  was  a  short,  four-waypoint 
route.  This  route  was  based  off  prominent  landmarks,  yet  still  required  the  pilot  to  make 
large  heading  changes. 

Figure  2  shows  the  intended  path  of  the  practice  route.  The  waypoints  are  the 
circles  labeled  one  through  four.  Adjacent  to  each  leg  is  a  navigational  totem, 
colloquially  referred  to  as  a  “doghouse,”  oriented  in  the  intended  direction  of  flight.  The 
doghouse  contains  four  pieces  of  navigational  data.  On  the  top  of  the  doghouse  is  the 
number  of  the  next  waypoint  on  the  intended  route.  Below  the  waypoint  entry  is  the 
intended  magnetic  heading  followed  by  the  distance  of  the  leg  in  nautical  miles  (nm). 
The  last  number  on  the  doghouse  is  time  it  takes  to  fly  the  given  leg  traveling  at  65  kts. 
“Doghouse”  annotations  were  not  provided  to  pilots  during  the  execution  of  the  test,  only 
in  the  prior  “map  study.” 
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Figure  2.  Map  of  the  practice  route  that  participants  used  for  map  study.  This  view 
shows  waypoints  1-4,  and  the  according  ‘doghouses’  which  give  the  pilot 
distance  and  timing  information  enroute. 

The  first  navigation  route  where  data  was  collected  was  designed  to  ease  the 
participants  into  the  experiment.  Figure  3  shows  the  intended  route  of  flight  for  the  route. 
For  each  scenario  the  participants  begin  the  route  exactly  on  the  first  waypoint  heading 
directly  to  the  second  waypoint.  Approaching  waypoint  two,  there  was  terrain  rising  off 
the  nose  of  the  aircraft  with  a  low  level  wash  to  the  right.  After  reaching  waypoint  two, 
participants  had  to  make  a  90-degree  turn  to  the  right  across  the  wash  to  the  entrance  of  a 
valley  at  waypoint  3.  The  valley  at  waypoint  3  can  be  misperceived  because  there  was 
similar  valley  that  leads  to  the  West  abeam  waypoint  three.  Participants  had  to  reference 
their  heading  to  make  sure  that  they  are  heading  down  the  correct  valley,  or  could  also 
notice  that  they  had  to  choose  the  valley  furthest  to  the  right.  The  leg  after  waypoint  3 
follows  the  valley  until  it  reaches  a  saddle  in  the  terrain  followed  by  a  drop  off  indicating 
waypoint  4.  The  route  from  waypoints  4  to  5  follow  low  level  terrain  to  the  furthest 
Northern  tip  of  the  higher  terrain  at  waypoint  5.  Heading  towards  waypoint  6  there  was  a 
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large  chance  of  misperceiving  terrain  if  the  participants  were  not  using  the  available 
visual  cues.  Along  this  leg,  several  valleys  resemble  the  one  at  waypoint  6.  Participants 
had  to  realize  that  waypoint  6  is  abeam  a  small  hill.  Waypoint  7  also  had  a  chance  to  be 
misperceived  because  of  a  lack  of  significant  terrain  in  the  area.  Participants  had  to  be 
cognizant  that  once  they  cross  a  valley  on  the  leg  from  6  to  7,  that  waypoint  7  is  just  on 
the  other  side.  Also  if  participants  miss  waypoint  7  they  have  to  recognize  the  limiting 
feature  of  the  large  flat  landscape  soon  after  waypoint  7.  Waypoint  8  should  be  easily 
recognized  if  the  participant  remained  close  to  track  because  it  is  the  only  large  hill  with 
little  terrain  around  it.  A  subject  matter  expert  believed  that  approximately  half  of  the 
participants  would  be  able  to  complete  this  route  without  any  help  from  the  proctor. 


Figure  3.  Map  of  navigation  route  1.  This  type  of  map  was  also  used  for  map  study. 

The  features  of  this  route  that  are  noteworthy  are  that  it  follows  easily 
marked  valleys.  Also,  note  that  there  are  no  large  heading  changes. 
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The  second  navigation  route  was  designed  to  task  saturate  the  pilots  to  a  level 
where  they  would  stray  off-course.  To  achieve  this,  navigation  legs  passed  through 
heavily  mountainous  terrain,  making  it  hard  for  the  pilots  to  maintain  good  navigational 
checkpoints.  This  route  also  required  turns  in  excess  of  90  degrees.  These  turns  also 
made  it  hard  for  the  pilots  because  it  required  them  to  continuously  search  for  new 
landmarks  along  the  route.  Figure  4  shows  the  intended  route  of  flight  for  the  second 
route.  In  this  route,  waypoint  2  was  identifiable  because  it  intersects  rising  terrain  in  the 
first  large  valley.  The  leg  from  2  to  3  follows  a  valley  up  to  a  ridge  at  waypoint  3.  This 
ridgeline  can  be  misperceived  if  the  participant  does  not  follow  the  track  because  of  other 
similar  high  terrain  in  the  area.  Misperception  can  then  lead  the  participant  to  track  down 
the  wrong  ridgeline  to  waypoint  4.  Waypoint  4  is  also  difficult  to  distinguish  because  of 
surrounding  similar  terrain.  Much  of  these  misperception  errors  due  to  similar  terrain 
were  solved  with  the  large  North-South  running  valley  which  contains  waypoint  5. 
Navigation  from  point  5  to  6  follows  the  valley  and  ends  at  the  entrance  to  the  valley  at 
waypoint  6.  After  waypoint  6  participants  had  to  make  a  left  hand  turn  to  waypoint  7  at 
the  entrance  to  a  Northeast  running  valley.  Waypoint  7  can  be  easily  misperceived 
because  there  is  a  similar  Northeast  mnning  valley  to  the  East  of  point  7.  Participants 
had  to  make  sure  that  they  took  the  first  valley  entrance  and  not  the  second.  Waypoint  8 
was  also  difficult  due  to  other  similar  high  terrain  in  the  vicinity.  After  hitting  waypoint 
8  at  the  mountain  peak,  participants  had  to  make  a  right  hand  turn  to  waypoint  9  to  the 
East.  Waypoint  9  is  located  along  a  valley  in  low  level  terrain  that  has  the  possibility  to 
be  overlooked,  but  was  limited  by  the  flat  terrain  to  the  East  of  this  point.  The  leg  from  9 
to  10  also  could  be  misperceived  by  a  similar  adjacent  valley  to  the  East  of  it.  It  was 
believed  that  only  a  small  percentage  of  the  participants  would  be  able  to  track  on-course 
throughout  this  route. 
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Figure  4.  Map  of  navigation  route  2.  Note  the  challenging  turns  throughout  and 
potential  for  overflying  WP  3  and  5  and  similar  valleys  at  WP  7  and  9 


The  two  autonavigation  routes  were  based  off  the  route  used  in  Sullivan  (2010) 
shown  in  Figure  5.  It  was  during  this  experiment  that  over  half  of  the  participants 
misperceived  the  valley  A  as  waypoint  6,  and  a  large  number  of  participants  had 
problems  locating  waypoint  9.  These  autonavigation  routes  were  designed  in  hopes  of 
determining  the  reason  why  participants  in  Sullivan  et  al.  (2010)  did  not  realize  they  were 
off-course  once  they  made  their  mistake. 
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Figure  5.  Map  of  the  navigation  routes  used  in  Sullivan  (2010). 


The  first  autonavigation  route,  Figure  6,  consisted  of  waypoints  3  through  9  of  the 
route  from  Sullivan  et  al.  (2010).  The  autonavigation  flight  path  closely  resembles  what 
some  of  the  participants  flew  in  Sullivan’s  experiment.  They  went  right  by  waypoint  4, 
and  flew  down  the  wrong  valley,  i.e.,  valley  A.  Instead  of  realizing  they  were  off  course, 
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they  continued  to  fly,  ending  up  to  the  North  of  the  intended  route.  This  thesis  looks  to 
understand  why  the  participants  were  misperceiving  this  segment  of  the  route. 


Figure  6.  Intended  (black)  and  actual  (yellow)  first  autonavigation  route 


The  second  autonavigation  route  resembles  the  second  half  of  Sullivan’s  (2010) 
route  (Figure  7).  The  actual  flight  path  of  route  also  resembles  what  some  of  the 
participants  of  Sullivan’s  (2010)  experiment  flew.  This  flight  path  is  considered  very 
difficult  to  follow  once  the  aircraft  turns  to  the  South  after  waypoint  4  because  pilots  are 
navigating  through  high  terrain  that  can  be  easily  misperceived. 
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Figure  7.  Map  of  the  second  auto  navigation  route.  Intended  (black)  and  actual 

(yellow)  second  autonavigation  route 


3.  Performance  Measures 

The  measures  collected  in  this  experiment  were  classified  into  two  categories, 
independent  and  dependent.  The  independent  variables  were  based  on  demographic  and 
experience,  while  the  dependent  variables  were  navigation  performance  and  confidence. 

a.  Data  Collection 

The  best  way  to  access  navigation  performance  is  to  combine  how 
accurately  the  participant  navigated  through  the  route  and  compare  it  to  their  confidence 
or  perception  (Sullivan,  2010).  Finding  the  accuracy  of  navigation  is  relatively  easy  to 
calculate,  and  was  one  of  the  measures  used  in  Sullivan  (2010).  On  the  opposite  end  of 


19 


the  spectrum  is  confidence.  Confidence  has  considerably  more  variance  than  accuracy, 
and  is  difficult  to  measure  a  benchmark  value.  It  is  likely  that  the  participants  would 
have  a  different  baseline  and  range  of  confidence  (Orban,  2008).  The  question  is  how  to 
best  measure  a  participant’s  confidence.  Ideally,  confidence  assessments  should  be 
collected  and  updated  constantly  throughout  the  navigation  route.  The  participant  could 
only  really  do  this  with  verbal  protocol.  The  biggest  challenge  is  that  there  is  poor 
standardization  between  participants.  Some  people  are  more  verbal  than  others,  and  it  is 
common  that  people  are  less  likely  to  say  when  they  are  wrong  then  when  they  are 
correct.  In  addition,  verbal  protocol  drops  off  when  a  participant  is  in  a  high  cognitive 
workload.  For  these  reasons,  Sullivan  used  post  route  questionnaires  to  determine 
confidence  throughout  the  route.  The  biggest  problem  with  the  post  route  questionnaire 
is  that  participants  quickly  forget  their  true  confidence  feelings  after  the  fact,  and  are 
likely  to  reduce  their  confidence  if  they  found  themselves  ultimately  getting  lost  during 
the  simulation. 

This  experiment  was  meant  to  collect  accurate  confidence  data  during  the 
navigation  process  without  the  disadvantages  of  verbal  protocol  or  post  route 
questionnaires.  One  of  the  hurdles  of  collecting  this  data  was  to  avoid  interrupting  the 
navigational  flow  for  the  participants.  This  means  that  large  breaks  were  minimized  in 
order  to  gather  the  most  realistic  data.  Break  times  were  reduced  by  integrating  a  user- 
friendly  confidence  application.  The  route  was  paused  at  20  to  40  second  intervals  and 
the  participant  was  asked  to  pinpoint  their  perceived  location  on  the  map  and  to  assign  a 
confidence  measure  to  their  perception.  Pause  break  times  were  also  minimized  by 
informing  participants  that  the  breaks  were  to  analyze  their  stream  of  consciousness  while 
navigating,  and  should  take  about  five  seconds  and  not  more  than  ten  seconds.  The  break 
times  were  not  to  be  used  to  readjust  their  position  by  getting  additional  time  to  look  at 
the  map  and  the  OTW  displays. 

b.  Independent  Variables 

Independent  variables  were  collected  from  the  background  questionnaire 
that  included  demographics  and  expertise.  Total  flight  hours,  overland  flight  hours, 
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participation  in  similar  past  experiments,  and  experience  with  low-level  and  desert  low- 
level  navigation  were  used  to  group  and  rank  the  participants  for  analysis. 

c.  Dependent  Variables 

There  were  two  major  dependent  variables  used  for  analysis;  confidence 
and  the  distance  from  the  actual  and  perceived  helicopter  position.  Confidence  was  based 
on  a  zero  to  100  percent  scale.  The  error  in  perceived  location  was  derived  from  the  great 
circle  distance  between  the  actual  latitude  and  longitude  position  of  the  aircraft  and  the 
participant’s  perceived  latitude  and  longitude.  The  following  equation  is  used  because  of 
the  Earth’s  curvature  (Gellert,  Gottwald,  Hellwich,  Kastner  &  Kiistner,  1989). 

d  =  arccos(sin  cpa  sin  (pp  +  cos  (pa  cos  (pp  cos  (Xa  -  Xp))  R 

where  d  =  Error  distance  (km) 

R  =  Earth’s  radius  at  the  Twentynine  Palms,  CA  area  =  6372.8  km 
(pa  =  Latitude  of  the  actual  aircraft  position  in  radians 
cpp  =  Latitude  of  the  perceived  aircraft  position  in  radians 
Xa  =  Longitude  of  the  actual  aircraft  position  in  radians 
Xp  =  Longitude  of  the  perceived  aircraft  position  in  radians 

Euclidian  distance  would  introduce  errors,  but  would  be  acceptable 
because  the  distances  involved  in  this  study  are  minimal  compared  to  the  curvature  of  the 
Earth. 


4.  Apparatus 

a.  Display 

The  participants  of  the  experiment  had  three  different  displays  on  two 
monitors.  The  out-the-window  display  was  located  directly  in  front  of  the  participant  and 
is  shown  in  Figure  8. 
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Figure  8.  Example  of  the  out  the  window  (OTW)  view  that  the  participants  saw  in  the 

experiment. 


The  map  used  for  this  experiment  is  the  1:50K  topographical  land  map 
(TLM).  This  type  of  map  is  used  in  the  training  commands  for  overland  navigation, 
terminal  area  maps  for  operational  missions,  and  orienteering.  This  map  is  also  a  good  fit 
for  our  experiment  because  the  grids  on  the  map  are  one  kilometer  in  length,  giving  the 
participants  a  quick  reference  on  distance;  we  believe  that  this  is  the  map  that  many  of 
our  subjects  would  choose  if  they  were  able.  1:50K  TLM  also  provides  in-depth  terrain 
elevation  data.  Because  cockpit  map  displays  vary  widely  among  the  different  helicopter 
platforms,  this  1:50  TLM  was  used  both  in  map  study  and  during  the  simulation.  The 
map  display  did  not  have  the  doghouse  information  on  it,  as  they  would  normally  have 
when  operating  in  theater,  forcing  the  participant  to  visually  navigate  through  the  route, 
and  use  less  emphasis  on  instrument  navigation.  The  cockpit  instrument  gauges  were 
selected  to  resemble  standard  military  gauges,  giving  the  participants  with  prior  aviation 
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experience  a  realistic  environment,  yet  kept  the  instrument  cluster  simple  enough  for 
participants  who  have  limited  experience  with  these  gauges.  Therefore,  the  experiment 
only  used  a  constant  running  clock,  heading  indicator,  barometric  altimeter,  and  radar 
altimeter  (note:  Altitude  was  fixed  at  150’  AGL  for  the  experiment).  Adding  all  of  the 
instrument  gauges  may  have  made  the  experiment  more  realistic,  but  that  could  also  give 
an  advantage  to  aviators  who  have  experience  with  that  cockpit  layout.  This  simplified 
instrument  cluster  was  adequate  to  complete  the  navigation  routes,  and  was  intended  to 
be  used  as  a  backup  rather  than  as  the  primary  navigation  tools. 


Figure  9.  Screen  shot  of  the  map  and  instrument  cluster  seen  by  the  participants 

during  the  experiment.  No  doghouses  are  provided  during  the  experiment. 

b.  Control 

As  mentioned  previously  the  PNAC  is  usually  the  one  who  is  in  charge  of 
navigation.  The  PNAC  give  verbal  navigational  commands  to  the  PAC.  This  brings 


23 


about  the  standardization  issue  for  the  experiment.  To  limit  the  discrepancies  of  verbal 
communication,  the  participant  would  be  required  to  navigate  and  fly  for  the  first  two 
routes.  To  allow  the  participant  to  cover  both  of  these  tasks,  the  aviating  task  had  to  be 
simplified.  This  simplification  was  achieved  by  using  autopilot  for  altitude  hold  and 
constant  airspeed.  This  altitude  was  set  to  150  feet  AGL  and  airspeed  was  set  to  65  kts. 
This  means  that  the  only  required  inputs  into  the  simulation  were  left  and  right  roll.  The 
roll  was  executed  using  a  joystick  resembling  a  helicopter  cyclic.  Navigation  using  this 
joystick  required  a  small  workload,  allowing  the  participant  to  concentrate  on  navigation 
and  not  pilotage. 


c.  Flight  Simulation 

The  setup  of  equipment  for  the  simulation  involved  a  cockpit  style  seat 
with  a  joystick  mounted  in  between  the  participant’s  legs.  The  joystick  placement,  look, 
and  control  are  similar  to  a  helicopter  cyclic.  The  joystick  allowed  the  participant  to  roll 
the  aircraft  in  the  lateral  direction  and  pitch  (up  and  down)  without  corresponding 
changes  in  altitude  and  airspeed,  and  gave  the  participant  the  ability  to  look  up  and  down. 
A  computer  mouse  was  mounted  to  the  right  side  of  the  seat  for  the  participant  to 
pinpoint  their  location  on  the  digital  map  and  adjust  their  confidence.  Directly  in  front  of 
the  participant  was  the  OTW  monitor  with  a  110cm  by  61cm  display.  The  OTW  monitor 
was  placed  four  feet  from  the  s  participant  and  covered  a  65-degree  field  of  view  (FOV). 
At  a  130  degree  right  offset  towards  the  participant  from  the  OTW  display  was  an  88.5cm 
by  50cm  display  used  for  the  map  and  instrument  panel.  The  display  map  was  a  1:50K 
topographical  land  map  (TLM),  the  same  type  as  the  participant  s  used  for  their  map 
study.  On  the  map  were  labeled  blue  circles  representing  numbered  waypoints.  The  map 
was  pointed  North  at  all  times  because  when  the  map  moved  automatically  with  the 
heading  the  aircraft  it  became  very  disorienting  to  the  pilot.  The  instrument  panel 
contained  a  heading  indicator  that  was  typical  of  Navy  H-60  displays.  Next  to  the 
heading  indicator  was  a  typical  barometric  altimeter  followed  by  a  radar  altimeter. 
Above  these  gauges  was  a  constant  running  digital- style  elapsed  time  clock.  This  clock 
started  when  the  simulation  program  started  and  did  not  stop,  even  when  the  simulation 
was  paused,  until  the  route  was  finished. 
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Figure  10.  Schematic  Diagram  of  Experiment  Setup,  Joystick  input  is  read  by  X-Plane 
and  initiates  the  Image  Generator,  which  then  displays  the  OTW,  map,  and 
instrument  outputs  to  the  participant  and  instructor. 
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Figure  11.  Equipment  layout  showing  the  location  of  participant,  instructor,  displays, 

and  software. 


The  software  used  to  run  the  simulation  was  Image  Generator,  Terrain  & 
Map  D8,  and  Data  Logger  by  Delta3D  and  OpenSceneGraph.  These  programs  used 
inputs  through  X-Plane  9.21rc2,  a  commercially  used  flight  simulator.  The  software 
coverts  the  X-Plane  data  into  the  OTW  and  map  views  based  on  the  participants  inputs. 
The  X-Plane  model  was  set  using  the  Fokker  Eindecker  E.I  airframe  with  a  modernized 
autopilot  and  GPS  flying  at  5,000  mean  sea  level  (MSL)  and  65  knots.  This  airframe  was 

25 


chosen  because  of  its  ability  to  fly  at  slow  speeds  and  not  stall.  The  image  generation  PC 
takes  the  X-Plane  position  information  but  changes  the  altitude  so  that  the  participants 
see  an  altitude  of  150  feet  AGL.  This  altitude  remained  fixed  throughout  the  route  and 
maintained  obstacle  clearance  in  the  mountainous  terrain.  Moving  the  joystick  up  and 
down  did  not  affect  the  pitch  of  the  aircraft,  but  did  allow  the  participant  to  look  up  and 
down.  The  roll  of  the  aircraft  was  completed  with  left  or  right  joystick  inputs.  This  put 
the  aircraft  in  coordinated  turns.  The  software  also  updated  the  instruments  to  correspond 
with  the  current  flight  profile. 

The  first  two  routes  required  participant  roll  inputs  to  navigate  through  the 
route,  while  the  last  two  routes  were  flown  on  autopilot.  The  autopilot  was  set  in  X-Plane 
to  follow  a  preselected  route.  The  participant  did  not  have  roll  control  but  had  the  ability 
to  look  up  and  down  on  the  screen  with  forward  and  back  control  movement. 

d.  Confidence  Application 

Confidence  App  Software  was  created  in  order  gain  useful  confidence 
output  data.  This  program  allowed  the  participant  to  click  where  they  perceived  to  be  on 
the  map  display.  After  the  participant  right  clicked  on  the  map,  a  red  dot  showed  on  the 
screen  and  a  confidence  scroll  bar  appeared.  This  confidence  bar  allowed  the  participant 
to  rate  how  confident  they  were  of  their  perceived  location.  This  bar  ranged  from  100, 
very  confident,  to  0,  very  lost.  After  the  route  was  complete,  the  software  also  created  a 
CSV  file  that  contained  the  elapsed  time  of  when  the  participant  made  his  location 
estimate,  the  actual  helicopter  latitude  and  longitude,  the  participant’s  estimated  latitude 
and  longitude,  and  the  participant’s  confidence  on  their  perceived  location. 
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Figure  12.  Snapshot  of  the  Confidence  App.  Red  dot  is  the  participant’s  perceived 

location  and  confidence  scroll  bar  from  0  to  100. 

5.  Procedures 

Participants  were  introduced  to  the  experimentation  lab  with  an  IRB  approved 
welcome  script  (Appendix)  that  notified  the  participant  of  the  focus  of  the  study,  brief 
overview  of  what  will  be  expected  out  of  them,  rules  of  the  lab,  and  the  voluntary  nature 
of  the  study.  The  participants  were  given  an  informed  consent  form,  also  listed  in 
Appendix,  to  read  and  sign.  The  form  reviewed  the  minimal  risks,  the  voluntary  nature, 
the  benefits,  and  confidentiality  of  participating  in  this  experiment.  After  the  informed 
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consent,  the  participant  was  given  a  questionnaire  relating  to  their  flying  experience  and 
background  (Appendix).  The  background  questionnaire  included  basic  demographics, 
familiarity  of  the  simulation-operating  environment,  experience  with  overland  navigation, 
flight  hours,  and  time  since  last  flight.  This  data  was  collected  to  help  group  the 
participants  for  analysis. 

Once  the  participant  completed  the  background  questionnaire,  they  were  given  a 
familiarization  of  the  experiment;  including  the  flight  parameters  of  the  helicopter,  what 
is  contained  on  the  video  screens,  joystick  control,  and  how  to  use  the  confidence  feature 
of  the  simulation.  Once  the  participant  seemed  comfortable  with  how  the  simulation 
would  run,  they  were  given  a  map  of  the  practice  route.  This  was  an  8x11  map  printed 
from  Falconview.  The  map  was  a  1:50K  TLM,  just  like  the  one  that  they  would  see  on 
the  monitor.  This  map  was  doghoused,  with  the  waypoint  number,  distance  in  nm,  time 
to  fly  the  leg,  and  total  elapsed  time.  This  paper  map  was  only  allowed  during  the  map 
study,  and  not  during  the  flight  portion  of  the  simulation.  The  participant  could  only  use 
the  map  on  the  monitor,  which  included  numbered  waypoints,  during  the  simulation.  The 
participants  were  given  unlimited  time  to  review  the  practice  map  before  flying  the 
simulation.  The  practice  simulation  was  four  waypoints  long  on  an  easy  route.  One  of 
the  main  objectives  of  the  practice  route  was  to  make  the  participant  comfortable  with  the 
flight  profile  and  monitor  views,  along  with  getting  a  solid  grasp  of  using  the  confidence 
program.  This  route  was  paused  roughly  every  30  seconds  for  the  participant  to  point  out 
their  perceived  location  on  the  map,  and  their  confidence  level.  The  participants  were 
given  some  navigation  assistance  from  the  proctor  if  they  were  lost.  Once  the  participant 
completed  the  route,  they  were  asked  if  they  were  comfortable  with  the  simulation  and 
programs.  If  necessary  they  were  allowed  to  have  extra  practice  flying  the  helicopter  if 
they  were  not  comfortable. 

After  the  completion  of  the  practice  route,  the  participants  were  tasked  with 
completing  four  navigation  routes  where  data  was  collected.  In  the  first  two  routes,  the 
participants  were  providing  roll  inputs  while  flying,  whereas  the  last  two  routes  were 
flown  on  autopilot.  The  first  two  routes  began  with  a  map  study  period  of  three  minutes, 
in  contrast  the  last  two  map  study  times  were  two  minutes.  Map  study  times  were  limited 
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to  provide  increased  difficulty  by  limiting  the  amount  of  headings  and  timings  on  the 
route,  and  to  keep  the  experiment  under  an  hour  in  duration.  The  last  two  map  study 
times  were  less  because  the  routes  were  shorter,  and  the  helicopter  was  on 
autonavigation,  reducing  the  task  load  on  the  participant.  Before  executing  the 
autonavigation  route,  the  participants  were  also  given  a  scenario.  In  this  scenario,  the 
participant  simulates  flying  with  a  new  pilot  in  the  squadron  who  is  responsible  for  the 
navigating  and  flying.  Both  the  new  pilot  and  the  participant  must  fly  in  an  area  where 
they  have  never  been.  The  new  pilot  is  supposed  to  follow  the  route,  but  there  is  a  chance 
that  they  can  get  off-track.  The  scenario  informs  the  participant  that  the  intended  route  is 
not  necessarily  what  the  new  pilot  will  fly.  Once  the  map  study  was  complete,  the 
participant  conducted  the  navigation  portion  of  the  simulation.  During  the  first  two 
simulations,  the  route  was  paused  about  every  40  seconds.  40  seconds  was  not  a  hard 
number  because  the  evaluator  wanted  to  minimize  pausing  during  turns.  Pausing  during 
turns  can  be  disorienting  to  participants,  and  it  is  hard  to  remember  the  amount  of  bank 
they  had  after  they  finished  the  pause.  During  the  second  two  simulations,  the  pause 
points  were  in  the  same  location  for  each  participant,  and  happened  between  20-40 
seconds.  Again,  these  pauses  occurred  during  level  flight.  After  the  completion  of  each 
of  the  navigation  routes,  the  participants  were  given  a  post  task  questionnaire  (Appendix). 
It  questioned  whether  the  participant  felt  they  strayed  off-course,  misperceived  terrain, 
and  asked  what  they  could  have  done  differently  to  remain  on-course. 

Once  all  four  routes  were  completed,  the  participants  were  given  one  final 
questionnaire  (Appendix).  This  questionnaire  covered  topics  on  why  they  believe  pilots 
get  lost,  what  they  do  if  they  sense  they  are  not  on-course,  and  what  they  think  their 
confidence  level  during  navigation  is.  This  questionnaire  allowed  for  participant 
grouping  based  off  similar  responses.  The  participants  were  asked  to  add  any  additional 
comments,  and  the  evaluator  asked  other  pertinent  questions  to  give  insights  on  why  they 
misperceived  terrain  on  the  route  and  confidence  levels. 
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6.  Participants 


To  participate  in  this  experiment  the  participant  needed  to  have  overland 
navigation  training.  Participants  for  this  study  were  recruited  from  the  Naval 
Postgraduate  School  student  body  and  faculty.  Recruitment  was  completed  through  an 
IRB  approved  E-mail  sent  to  Operations  Research  and  Modeling,  Virtual  Environments 
and  Simulation  (MOVES)  students.  In  addition,  recruitment  was  done  through  word  of 
mouth  and  using  past  experiment  participants. 

There  were  a  total  of  15  male  and  female  participants  ranging  from  27  to  41  years 
of  age,  with  an  average  of  36  years  with  a  standard  deviation  of  4.8.  Total  flight  hours 
ranged  from  0  to  2,500  with  an  average  of  1,431  and  a  standard  deviation  of  803.5.  Total 
overland  hours  ranged  from  0  to  2,000  with  an  average  of  870  and  a  standard  deviation  of 
634.2.  There  were  eleven  U.S.  Navy,  four  U.S.  Army  participants,  and  one  Hellenic  Air 
Force  participant. 
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III.  ANALYSIS  OF  EXPERIMENTAL  DATA 


A.  VARIABLE  DEFINITAION 

This  section  covers  the  variables  that  were  used  for  data  analysis  and  their 
definitions. 

1.  Confidence  (CONF) 

Pilots’  confidence  was  self-reported  using  the  Confidence  App,  i.e.,  participants 
rated  their  navigation  confidence  from  0  to  1  for  each  pause  point.  The  CONF  is  defined 
as  confidence  measurement  between  0  and  1,  where  0  indicates  the  lowest  confidence  and 
1  the  highest  confidence.  CONF_BIN  is  a  variation  of  CONF  coerced  into  a  binary 
variable.  The  CONF_BIN  is  defined  as 

CONF_BIN  =  High  if  CONF  >  0.5 
Low  if  CONF  <0.5 

The  threshold  of  0.5  was  chosen  for  the  CONF_BIN  variable  because  it  was  the 
numerical  midpoint  of  the  CONF  range.  This  midpoint  was  easily  defined  on  the 
Confidence  App,  making  it  a  likely  division  between  high  confidence  and  low 
confidence.  If  a  participant  believed  there  was  a  good  chance  their  perceived  location  is 
not  close  to  the  actual  location  they  would  not  choose  a  confidence  level  over  0.5. 

2.  Error  (Correctness) 

To  measure  the  navigation  performance  of  the  participant,  error  distance  was 
solved.  This  error  distance  was  defined  as  the  great  circle  distance  (Gellert,  1989)  from 
where  the  subject  perceived  they  were  compared  to  where  they  actually  were  during  the 
pause  points. 

ERROR1  =  great  circle  distance  between  perceived  and  actual  location  in  km 
=  arccos(sin  cpa  sin  (pp  +  cos  (pa  cos  (pp  cos  (Xa  -  Xp))  R 

where  R  =  Earth’s  radius  at  the  Twentynine  Palms,  CA  area  =  6372.8  km 
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(pa  =  Latitude  of  the  actual  aircraft  position  in  radians 
tpp  =  Latitude  of  the  perceived  aircraft  position  in  radians 
%a  =  Longitude  of  the  actual  aircraft  position  in  radians 
XP  =  Longitude  of  the  perceived  aircraft  position  in  radians 


Similarly,  the  second  type  of  error  that  was  calculated  was  the  distance  between 
where  the  participant  perceived  they  were  compared  to  the  intended  route  of  flight. 

ERROR2  =  great  circle  distance  between  perceived  and  planned  location  in  km 
=  arccos(sin  (p;  sin  cpp  +  cos  (p;  cos  (pp  cos  (Xi  -  Xp))  R 

where  R  =  Earth’s  radius  at  the  Twentynine  Palms,  CA  area  =  6372.8  km 
(pi  =  Latitude  of  the  planned  aircraft  position  in  radians 
(pp  =  Latitude  of  the  perceived  aircraft  position  in  radians 
Xi  =  Longitude  of  the  planned  aircraft  position  in  radians 
Xp  =  Longitude  of  the  perceived  aircraft  position  in  radians 

The  next  derived  variable,  NAV,  took  the  ERROR  1  distance  and  turned  it  into  an 
indicator  variable  to  state  whether  the  participant  stayed  within  a  certain 
threshold/boundary.  Pilots  were  instructed  to  stay  within  .5  km  of  the  route;  we  buffered 
this  to  be  .75  km. 

NAV  =  On-track  if  ERROR  1  <  0.75  km 
Off-track  if  ERROR1  >  0.75  km 

The  0.75  km  distance  for  obtaining  the  NAV  variable  was  used  because 
participants  were  told  prior  to  their  navigation  tasks  that  they  should  be  confident  in  their 
perceived  location  if  they  were  within  0.5km  of  their  actual  location.  The  0.75  km  gave 
the  subjects  an  additional  0.25  km  error  distance  because  it  is  difficult  for  pilots  to 
recognize  if  they  fell  within  the  0.5  km  distance  while  navigating.  This  additional  error 
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distance  also  helped  to  affirm,  without  any  doubt,  that  the  participant  has  the  wrong 
perception  of  their  location. 

3.  Experience 

The  study  also  conducted  exploratory  analysis  on  the  effect  of  pilot  experience. 
This  analysis  required  defining  variables  for  pilot  flight  hours,  expertise,  familiarity  with 
low  level  desert  navigation,  and  whether  the  subject  participated  in  a  similar  past  study 
and  consequently  may  have  experienced  a  practice  effect.  These  variables  are  shown 
below. 

EXPERTISE  =  Expert  if  participants  total  number  of  flight  hours  >  1,000 
Novice  if  participants  total  number  of  flight  hours  <  1,000 

PROFICIENT  =  Yes,  if  the  participant  answered  Somewhat,  Considerable,  or 
Extensive  for  their  experience  with  low  level  desert  navigation 

No  if  the  participant  answered  Very  Little  or  None 

PAT_EXP  =  Yes,  if  the  subject  participated  in  a  similar  past  study 
No,  if  they  did  not  participate  in  a  past  study 

B.  COMPREHENSIVE  NAVIGATION  PERFORMANCE 

The  first  analysis  of  the  data  related  the  experiment  output  to  the  Sullivan’s 
(2010)  modified  SDT  matrix  for  assessing  navigation  skills  (Table  1).  The  experiment 
focused  on  the  “dangerous”  quadrant  as  defined  in  Chapter  I;  the  pilot  does  not  realize 
that  they  are  lost.  Table  3  shows  the  confidence  versus  navigational  error  using  the 
CONF_BIN  and  NAV  variables. 
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Assessing  Navigation 

CONF_BIN 

Performance 

Low 

High 

Grand  Total 

Off¬ 

7.67% 

26.98% 

34.65% 

NAV 

track 

(22.14%) 

(77.86%) 

(100%) 

On-track 

6.98% 

58.37% 

65.35% 

(10.68%) 

(89.32%) 

(100%) 

Grand  Total 

14.65% 

85.35% 

100% 

Table  3.  Matrix  of  experimental  navigation  performance  relating  CONF_BIN  and 
NAV.  Percentages  in  parentheses  are  calculated  based  on  Off-track  or  On- 
track  NAV  respectively.  Of  interest  is  the  bolded  area,  corresponding  to  off 
track  and  high  confidence,  or  the  “dangerous”  quadrant. 

Table  3  shows  that  58.37%  of  the  time  during  the  navigation  participants  were 
On-track  and  had  a  corresponding  high  confidence  level.  This  table  also  shows  that  only 
6.98%  of  time  pilots  had  low  confidence  yet  still  were  considered  On-track.  These 
percentages  reflect  that  the  subject  is  highly  unlikely  to  misperceive  their  location  when 
on-track,  but  the  problem  arises  when  the  participants  were  Off-track.  Subjects  were  off¬ 
track,  yet  still  highly  confident  26.98%  of  the  time  during  the  navigation.  This  indicates 
subjects  were  highly  confident  about  their  navigation  performance  77.86%  of  the  time 
when  they  were  off-track.  The  misperception  error  is  about  3.5  times  greater  than  correct 
perception  when  a  pilot  is  off  track.  This  relates  to  the  dangerous  section  of  the  matrix 
where  pilots  are  lost  and  do  not  know  it,  and  this  is  the  second  largest  navigational  state 
of  the  experiment  among  four  navigational  states.  It  is  also  in  this  area  where  mission 
failure  and  mishaps  occur  due  to  incorrect  navigation.  Figure  13  shows  the  breakdown  of 
navigation  performance  matrix. 
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Figure  13.  Pie  chart  of  comprehensive  navigation  performance  matrix.  Notice  Off¬ 
track  and  High  CONF  is  second  highest  performance  state,  and  greater  than 
40%  of  participants  were  away  from  ‘On  track,  high  confidence,’ 
suggesting  that  overland  navigation  is  a  complex  task. 


Table  3  is  further  broken  down  to  see  if  different  routes  result  in  different 
perception  output.  Table  4  compares  the  route  and  CONF_BIN  and  NAY. 


Assessing  Navigation  Performance 

Route 

1 

2 

3 

4 

Average 

NAV  and 
CONF_BIN 

Off-track 

26.6% 

28.4% 

35.6% 

34.8% 

30.9% 

High  Confidence 

19.5% 

(73.3%) 

21.3% 

(75.0%) 

31.7% 

(88.9%) 

25.8% 

(74.2%) 

24.2% 

(78.2%) 

Low  Confidence 

7.1% 
(26.7  %) 

7.1% 

(25.0%) 

4.0% 

(11.1%) 

9.0% 

(25.8%) 

6.7% 

(21.8%) 

On-track 

73.5% 

71.7% 

64.4% 

65.2% 

69.1% 

High  Confidence 

65.5% 

(89.2%) 

65.4% 

(91.2%) 

55.5% 

(86.2%) 

56.2% 

(86.2%) 

61.2% 

(88.5%) 

Low  Confidence 

8.0% 

(10.8%) 

6.3% 

(8.8%) 

8.9% 

(13.8%) 

9.0% 

(13.8%) 

7.9% 

(11.4%) 

Total 

100.0% 

100.0% 

100.0% 

100.0% 

100.0% 

Table  4.  Matrix  of  experimental  navigation  performance  for  each  route  comparing 
navigation  performance  and  confidence,  the  auto  navigation  routes  (3  and  4) 
had  a  lower  percentage  of  route  correctness,  73.5%  and  71.7%  for  routes  1 
and  2  versus  64.4%  and  65.2%  for  routes  3  and  4,  respectively. 
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Table  4  shows  that  the  confidence  and  correctness  for  each  route  align  with  the 
overall  breakout.  The  most  interesting  fact  that  Table  4  shows  is  that  the  auto  navigation 
routes  (3  and  4)  had  a  lower  percentage  of  route  correctness,  73.5%  and  71.7%  for  routes 
1  and  2  versus  64.4%  and  65.2%  for  routes  3  and  4  respectively.  The  participants 
misperceive  their  location  more  frequently  when  control  inputs  were  not  required  from 
them.  Some  explanations  for  this  could  be  due  to  complacency,  and/or  experiment 
fatigue.  During  the  auto  navigation  routes,  participants  seemed  to  be  more  relaxed  during 
the  navigation  and  map  study.  Participants  were  less  likely  to  be  actively  tracking  the 
course,  which  lead  them  to  believe  that  the  aircraft  was  heading  on  course.  This  type  of 
complacency  is  common  in  multi-piloted  aircraft  and  can  be  attributed  to  mishaps.  Also 
noteworthy  is  the  fact  that  route  3  had  the  highest  percentage  of  time  in  the  “dangerous” 
quadrant.  Order  effects  may  explain  why  route  3  was  higher  than  the  other  routes.  Route 
3  could  be  higher  than  route  1  and  2  because  route  3  was  the  first  time  the  participant 
dealt  with  autonavigation.  Additionally,  route  3  could  also  be  higher  than  route  4 
because  they  pilot  realized  at  the  end  of  route  3  that  the  autonavigation  did  not  follow  the 
intended  route  of  flight,  making  the  CONF  on  route  4  less  than  3.  This  would  correspond 
to  a  lower  amount  of  time  in  the  “dangerous”  quadrant. 

Route  Performance  by  Route 

80% 

70% 
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50% 

40% 

30% 

20% 

10% 

0% 


Figure  14.  Column  graph  of  navigation  performance  by  route 
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c. 


HYPOTHESIS  TESTING 


We  used  Spearman’s  rank  correlation  coefficient  to  determine  statistical 
dependence  between  variables  for  the  first  two  hypothesis  tests.  Spearman’s  rank 
correlation  is  a  nonparametric  measure,  and  is  commonly  denoted  as  p  (rho).  Spearman’s 
rank  correlation  coefficient  is  defined  as  Pearson’s  correlation  coefficient  with  ranked 
variables  (Myers  &  Well,  2003)  and  is  given  by 

6  Yd? 

p  =  1 - 

n(n~  -1) 

where  dj  =  the  difference  in  rank  between  confidence  and  error  distance 

n  =  the  cumulative  number  of  pause  points  examined  for  each  route 

The  significance  level,  a,  was  set  at  0.05.  To  avoid  confusion,  for  the  remainder 
of  this  paper,  when  we  refer  to  p ,  we  mean  Spearman’s  measure. 

1.  Hypothesis  1:  Navigation  Error  and  Confidence  is  Negatively 
Correlated,  i.e.,  as  Navigation  Error  Increases,  Corresponding 
Confidence  Decreases. 

Hypothesis  1  was  examined  for  each  route.  The  average  CONF  and  ERROR  1 
was  computed  for  each  route,  and  then  these  averages  were  tested  for  correlation.  Below 
are  the  null  and  alternative  hypotheses. 

H0 1 1 :  There  is  a  positive  or  no  correlation  between  CONF  and  ERROR1  for  the 
first  route,  i.e.,  pi>0 

Hai.it  There  is  a  negative  correlation  between  CONF  and  ERROR1  for  the  first 
route,  i.e.,  pi<0 


Hoi .2-  There  is  a  positive  or  no  correlation  between  CONF  and  ERROR1  for  the 
second  route,  i.e.,  p2>0 

Hai.2:  There  is  a  negative  correlation  between  CONF  and  ERROR1  for  the  second 
route,  i.e.,  p2<0 
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Hoi .3:  There  is  a  positive  or  no  correlation  between  CONF  and  ERROR1  for  the 
third  route,  i.e.,  p3>0 

Hai.3:  There  is  a  negative  correlation  between  CONF  and  ERROR1  for  the  third 
route,  i.e.,  p3<0 


H01.4:  There  is  a  positive  or  no  correlation  between  CONF  and  ERROR1  for  the 
fourth  route,  i.e.,  p4>0 

Hai.4:  There  is  a  negative  correlation  between  CONF  and  ERROR1  for  the  fourth 
route,  i.e.,  p4<0 


Table  5  shows  Spearman’s  rank  correlation  for  each  route. 


Pi 

P2 

P3 

P4 

-0.150 

0.158 

0.036 

-0.354 

Table  5.  Spearman’s  rank  correlation  coefficient  relating  CONF  and  ERROR1  for 
each  route.  Notice  that  there  are  not  statistically  significant  values  (all  p- 

values  >0.1) 

Using  a  one-tailed  test  of  Student’s  t  distribution  at  a  =  0.05  level,  we  fail  to  reject 
Hoi,  Ho2.  H03,  and  H04.  Thus,  we  cannot  claim  that  navigation  error  and  confidence  is 
negatively  correlated  for  routes  1  through  4.  This  is  a  surprising  result.  One  would  think 
that  there  should  be  a  high  correlation  between  confidence  and  perceived  location. 

This  result  has  several  implications  to  real  world  navigation.  When  a  pilot  is 
confident  in  their  location,  it  does  not  necessarily  mean  that  they  on  course.  Knowledge 
of  this  fact  could  help  reduce  the  amount  of  complacency  of  the  crewmembers  who  are 
not  actively  navigating.  This  result  can  also  be  used  a  wake-up  call  for  the  navigating 
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pilot.  They  might  reduce  their  confidence  level  that  they  are  on  track  causing  them  to  be 
more  proactive  in  the  cockpit  and  help  notice  potential  dangers  if  they  are  not  following 
the  route  correctly. 
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Figure  15.  CONF  versus  ERROR1  for  each  route.  Note  there  looks  to  be  little 

correlation  between  the  variables. 


2.  Hypothesis  2:  Distance  Between  Perceived  Location  and  Intended 
Route  is  Negatively  Correlated  to  Confidence,  i.e.,  as  the  Distance 
Between  the  Perceived  Location  and  Intended  Route  Increases, 
Corresponding  Confidence  Decreases. 

Ho2.i:  There  is  a  positive  or  no  correlation  between  CONF  and  ERROR2  for  the 
first  route,  i.e.,  pi>0 
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Ha2.i:  There  is  a  negative  correlation  between  CONF  and  ERROR2  for  the  first 
route,  i.e.,  pi<0 


H02.2:  There  is  a  positive  or  no  correlation  between  CONF  and  ERROR2  for  the 
second  route,  i.e.,  p2>0 

Ha2.2-  There  is  a  negative  correlation  between  CONF  and  ERROR2  for  the  second 
route,  i.e.,  p2<0 


H02.3:  There  is  a  positive  or  no  correlation  between  CONF  and  ERROR2  for  the 
third  route,  i.e.,  p3>0 

Ha2.3:  There  is  a  negative  correlation  between  CONF  and  ERROR2  for  the  third 
route,  i.e.,  p3<0 

H02.4:  There  is  a  positive  or  no  correlation  between  CONF  and  ERROR2  for  the 
fourth  route,  i.e.,  p4>0 

Ha2.4:  There  is  a  negative  correlation  between  CONF  and  ERROR2  for  the  fourth 
route,  i.e.,  p4<0 


Table  6  shows  Spearman’s  rank  correlation  for  each  route. 


Pi 

P2 

p3 

P4 

-0.650* 

-0.266 

0.011 

-0.596* 

t  p<10,  *  p  <  .05,  **  p  <  .01,  ***  p  <  .001 

Table  6.  Spearman’s  rank  correlation  coefficient  relating  CONF  and  ERROR2  for 

each  route 

Using  a  one-tailed  test  of  Student’s  t  distribution  at  a  =  0.05  level,  we  reject  H0i 
and  H()4  Thus,  we  can  claim  that  the  distance  between  perceived  location  and  intended 


40 


route  and  confidence  is  negatively  correlated  for  routes  1  and  4.  This  means  that  the 
participant  has  high  confidence  when  they  believe  they  are  close  to  the  intended  route  for 
routes  1  and  4  regardless  of  their  actual  closeness.  This  result  shows  that  there  is 
evidence  of  biased  visual  perception  favoring  their  intended  location. 
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Figure  16.  CONF  versus  ERROR2  for  each  route.  There  looks  to  be  some  correlation 

between  the  variables. 


Adding  to  the  ERROR2  and  CONF  correlation  Table  7  shows  the  average 
ERROR  1  and  ERROR2  distance  for  each  route. 


Route 

Avg  ERROR1 

Avg  ERROR2 

%  Difference 

1 

0.5304 

0.1311 

404.58% 

2 

0.6337 

0.1805 

351.08% 

3 

0.7411 

0.2525 

293.50% 

4 

0.704 

0.2122 

331.76% 
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Table  7.  Average  ERROR  1  and  ERROR2  for  each  route  and  the  percent  difference 

in  the  averages. 
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Figure  17.  Column  graph  showing  the  difference  between  ERROR  1  and  ERROR2 

Table  7  and  Figure  17  show  how  the  participants  were  biased  towards  the 
intended  route  of  flight.  ERROR1  was  about  three  and  a  half  times  larger  than  the 
corresponding  ERROR2.  Completing  a  paired  t-test  between  ERROR1  and  ERROR2, 
results  in  a  t-stat  of  1 1 .4  and  a  corresponding  p-value  of  p  <  0.001.  This  means  that  when 
confused,  pilots  had  a  tendency  to  perceive  they  were  where  they  were  supposed  to  be, 
and  not  defaulting  to  being  further  off  track  than  they  actually  are.  This  result  is  again 
helpful  for  pilots,  because  if  they  know  that  there  is  tendency  to  overestimate  their 
navigational  skills,  they  might  reduce  their  estimation  and  corresponding  confidence. 


3.  Hypothesis  3:  Perception  Error  Increases  the  Longer  the  Participant 
Navigates,  i.e.,  the  Distance  Between  Perceived  Location  and  Actual 
Location  is  Larger  at  the  End  of  the  Route  Compared  to  the 
Beginning. 

Ho3.i:  ERROR1  at  the  end  of  the  route  for  the  first  route  is  smaller  or  equal  to 
ERROR  1  at  the  beginning  of  route  one. 
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Ha31:  ERROR  1  at  the  end  of  the  route  for  the  first  route  is  larger  than  ERROR  1 
at  the  beginning  of  route  one. 


H03.2:  ERROR1  at  the  end  of  the  route  for  the  first  route  is  smaller  or  equal  to 
ERROR  1  at  the  beginning  of  route  two. 

Ha3.2:  ERROR1  at  the  end  of  the  route  for  the  first  route  is  larger  than  ERROR1 
at  the  beginning  of  route  two. 


H03  3:  ERROR1  at  the  end  of  the  route  for  the  first  route  is  smaller  or  equal  to 
ERROR  1  at  the  beginning  of  route  three. 

Ha33:  ERROR1  at  the  end  of  the  route  for  the  first  route  is  larger  than  ERROR1 
at  the  beginning  of  route  three. 


H()3.4:  ERROR1  at  the  end  of  the  route  for  the  first  route  is  smaller  or  equal  to 
ERROR  1  at  the  beginning  of  route  four. 

Ha3.4:  ERROR1  at  the  end  of  the  route  for  the  first  route  is  larger  than  ERROR1 
at  the  beginning  of  route  four. 


Table  8  shows  the  t  statistics  for  each  route. 
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t3.i(14) 

13.2(14) 

13.3(14) 

13.4(14) 

-2.067 

p-value  0.0260* 

-3.150 

p-value  0.0028** 

-7.708 

p-value  1.05E-06*** 

-2.816 

p-value  0.0069** 

t  p<10,  *  p  <  .05,  **  p  <  .01,  ***  p  <  .001 

Table  8.  t-  statistics  comparing  ERROR1  in  the  beginning  of  the  route  and  ERROR1 
at  the  end  of  the  route  for  each  route,  a  =  0.05  We  reject  the  null  hypothesis 
in  each  case,  showing  that  ERROR1  increased  at  the  end  of  the  route. 


Table  8  supports  the  hypothesis  that  there  is  a  statistically  significant  difference  in 
ERROR1  at  the  beginning  of  the  route  and  ERROR1  at  the  end  of  the  route.  This 
suggests  that  perception  error  gets  larger  the  longer  the  participant  flies.  This  result  can 
help  pilots  realize  that  they  might  want  to  reevaluate  their  perceived  location  the  further 
along  the  route  they  are,  and  reduce  their  corresponding  confidence  in  their  location. 

4.  Hypothesis  4:  Confidence  Decreases  the  Longer  the  Participant 
Navigates,  i.e.,  the  Participants  Confidence  Level  is  Larger  at  the 
Beginning  of  the  Route  Compared  to  the  End. 

Ho4. i :  CONF  at  the  end  of  the  route  for  the  first  route  is  smaller  or  equal  to  CONF 
at  the  beginning  of  route  one. 

Ha4.i:  CONF  at  the  end  of  the  route  for  the  first  route  is  larger  than  CONF  at  the 
beginning  of  route  one. 


H04.2:  CONF  at  the  end  of  the  route  for  the  first  route  is  smaller  or  equal  to  CONF 
at  the  beginning  of  route  two. 

Ha4  2:  CONF  at  the  end  of  the  route  for  the  first  route  is  larger  than  CONF  at  the 
beginning  of  route  two. 
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Ho4.3:  CONF  at  the  end  of  the  route  for  the  first  route  is  smaller  or  equal  to  CONF 
at  the  beginning  of  route  three. 

Ha4.3:  CONF  at  the  end  of  the  route  for  the  first  route  is  larger  than  CONF  at  the 
beginning  of  route  three. 


H04  4:  CONF  at  the  end  of  the  route  for  the  first  route  is  smaller  or  equal  to  CONF 
at  the  beginning  of  route  four. 

Ha4.4:  CONF  at  the  end  of  the  route  for  the  first  route  is  larger  than  CONF  at  the 
beginning  of  route  four. 

Table  9  shows  the  t-test  for  each  route. 


U.i(14) 

14.2(14) 

14.3(14) 

14.4(14) 

3.105 

p-value  0.00234** 

2.401 

p-value  0.0120* 

2.310 

p-value  0.0145* 

2.902 

p-value  0.00401** 

t  p<.10,  *  p  <  .05,  **  p  <  .01,  ***  p  <  .001 

Table  9.  Student  t  test  relating  CONF  in  the  beginning  of  the  route  and  CONF  at  the 

end  of  the  route  for  each  route,  a  =  0.05 


Table  9  supports  the  hypothesis  that  there  is  a  statistically  significant  difference  in 
CONF  at  the  beginning  of  the  route  and  CONF  at  the  end  of  the  route.  This  suggest  that 
the  longer  the  participant  navigates  along  a  route,  their  corresponding  confidence  gets 
lower.  This  result  follows  along  with  hypothesis  3,  that  the  perceived  error  appears  to 
increase  the  longer  the  participant  navigates.  Pilots  CONF  is  reducing  with  an  increasing 
ERROR  1.  Although  there  is  no  correlation  between  CONF  and  ERROR  1,  there  is  a 
trending  effect  of  CONF  getting  lower  further  into  the  route  while  ERROR  1  is 
increasing. 
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D.  EXPLORATORY  ANALYSIS  ON  EXPERTISE  EFFECT 


Analysis  was  done  to  see  if  there  was  any  effect  on  the  results  due  to  experience. 
This  experience  was  broken  down  into  two  categories:  expertise  effect  and  learning 
effect. 


1.  Expertise  Effect 

To  check  the  difference  between  expert  and  novice  pilots,  analysis  comparing  the 
effect  of  total  flight  hours  (EXPERTISE)  and  low-level  desert  navigation  experience 
(PROFICIENT). 


a.  EXPERTISE  ( Total  Flight  Hours ) 

CONF,  ERROR1,  and  ERROR2  were  analyzed  for  EXPERTISE  on  each 
route.  The  experiment  included  11  Expert  and  4  Novice  participants.  Figure  18  shows 
the  averages  of  each  of  the  variable  with  an  error  bar  of  a  95%  confidence  level. 
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Figure  1 8 .  EXPERTISE  effect  on  CONF,  ERROR  1 ,  and  ERROR2  for  each  route. 

Error  bars  are  for  a  95%  confidence  level. 


Figure  18  shows  that  there  is  overlap  in  confidence  levels  for  every 
variable  on  every  route.  This  means  that  EXPERTISE  does  not  have  an  effect  on  CONF, 
ERROR  1  and  ERROR2.  To  further  examine  the  relationship  between  EXPERTISE  and 
the  variables,  Student  t  tests  between  expert  and  novice  were  conducted  for  each  route 
and  shown  in  Table  10  for  an  a  =  0.05. 
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Route 

1 

2 

3 

4 

CONF 

t3 

0.363 

-0.0195 

-0.111 

0.141 

p-value 

0.735 

0.986 

0.917 

0.895 

ERROR1 

t3 

-0.816 

-0.0898 

0.547 

-0.383 

p-value 

0.475 

0.932 

0.608 

0.727 

ERROR2 

t3 

1.34 

-0.538 

-1.169 

-0.870 

p-value 

0.203 

0.610 

0.327 

0.433 

t  p<-10,  *  p  <  .05,  **  p  <  .01,  ***  p  <  .001 


Table  10.  Student  t  test  on  EXPERTISE  for  CONF,  ERROR  1,  and  ERROR2  for  each 
route.  We  do  not  reject  the  null  hypothesis  in  any  case  here,  suggesting  the 
EXPERTISE  does  not  have  an  effect  on  the  experiment. 


Table  10  shows  that  there  is  no  statistically  significant  correlation  on 
EXPERTISE.  This  also  means  that  there  is  also  no  effect  on  Expertise  for  the  experiment 
using  t-tests. 

The  final  check  to  see  if  EXPERTISE  had  an  effect  on  the  data  was  to  see 
if  Experts  and  Novices  were  in  the  “dangerous”  quadrant  for  a  similar  amount  of  time. 
Table  1 1  shows  the  matrix  breakdown  of  EXPERTISE. 


NAV/CONF BIN 

Expert 

Novice 

Average 

On-track 

63.61% 

70.18% 

65.35% 

High 

58.54% 

48.25% 

55.81% 

Low 

5.06% 

21.93% 

9.53% 

Off-track 

36.39% 

29.82% 

34.65% 

High 

28.16% 

23.68% 

26.98% 

Low 

8.23% 

6.14% 

7.67% 

Total 

100.00% 

100.00% 

100.00% 

Table  11.  Participant  navigation  performance  matrix  based  on  EXPERTISE.  The 
highlighted  portion  shows  that  Expert  and  Novice  participants  were  in  the 
“dangerous”  quadrant  a  similar  amount  of  time 


Table  11  shows  that  Experts  were  actually  in  the  “dangerous”  quadrant 
more  than  the  Novice  pilots,  yet  both  were  fairly  close.  This  result  is  surprising,  and  can 
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again  be  helpful  for  crew  coordination  purposes.  Although  a  pilot  may  have  a  lot  of 
flight  hours,  it  does  not  mean  that  their  perception  is  better  than  Novice  pilots. 


b.  PROFICIENT  (Experience  with  Low-Level  Desert  Navigation ) 

Analysis  on  the  effect  of  PROFICIENT  was  examined  just  as 
EXPERTISE.  This  analysis  looks  to  show  if  there  is  any  correlation  between  pilots  who 
have  experience  with  low-level  desert  navigation  and  those  who  have  very  little  to  none. 
There  were  seven  participants  who  were  PROFICIENT  and  eight  who  were  not.  Figure 
19  shows  the  average  breakdown  of  CONF,  ERROR1  and  ERROR2  for  each  route  based 
on  PROFICIENT. 
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Figure  19.  Column  graphs  of  PROFICIENT  and  average  CONF,  ERROR1,  and 
ERROR2  for  each  route.  Error  bars  are  based  on  95%  confidence  level. 
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Figure  19  shows  that  there  is  overlap  in  confidence  levels  for  all  three 
variables  for  each  route  except  for  CONF  on  route  1.  This  shows  that  PROFICIENT  has 
little  effect  on  participant’s  experimental  performance.  To  further  look  into  the  effect  of 
PROFICIENT  Student  t  tests  were  completed  and  shown  in  Table  12. 


Route 

1 

2 

3 

4 

CONF 

t7 

-1.611 

-0.772 

-0.808 

-1.134 

p-value 

0.146 

0.462 

0.442 

0.289 

ERROR1 

t7 

1.785 

0.577 

-0.985 

-1.106 

p-value 

0.0976f 

0.574 

0.344 

0.292 

ERROR2 

t7 

2.675 

1.180 

0.637 

-0.674 

p-value 

0.0191* 

0.261 

0.536 

0.513 

t  p<.10,  *  p  <  .05,  **  p  <  .01,  ***  p  <  .001 

Table  12.  Student  t-test  for  the  correlation  between  PROFICIENT  for  CONF, 

ERROR  1,  and  ERROR2  for  each  route. 


Table  12  shows  that  there  is  little  statistically  significant  differences 
between  PROFICIENT  pilots.  Although  not  significant,  it  is  interesting  to  note  that  for 
every  route,  CONF  was  lower  for  pilots  who  were  PROFICIENT.  This  may  at  first  seem 
a  little  counterintuitive,  because  one  would  think  that  they  would  be  more  confident  with 
their  abilities.  However,  they  could  have  been  less  confident  because  they  know  how 
easy  it  is  to  misperceive  terrain  in  the  desert  environment.  The  only  statistically 
significant  difference  is  ERROR2  for  route  1.  This  could  show  that  PROFICIENT  pilots 
had  less  bias  to  the  intended  route  of  flight  for  route  1.  Because  this  is  the  only 
significant  result,  and  that  ERROR2  is  not  significant  for  the  other  routes  (ERROR2  is 
actually  higher  for  PROFICIENT  pilots  in  route  4),  PROFICIENT  no  effect  on  this 
experiment. 

PROFICIENT  was  then  checked  to  see  if  it  had  a  correlation  effect  on  the 
“dangerous”  quadrant  of  the  performance  matrix.  Table  13  shows  the  performance 
matrix  breakdown  based  on  PROFICIENT. 
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PROFICIENT 

NAV/ 

CONF BIN 

Yes 

No 

Total 

On-track 

65.83% 

64.94% 

65.35% 

High 

49.25% 

61.47% 

55.81% 

Low 

16.58% 

3.46% 

9.53% 

Off-track 

34.17% 

35.06% 

34.65% 

High 

22.61% 

30.74% 

26.98% 

Low 

11.56% 

4.33% 

7.67% 

Total 

100.00% 

100.00% 

100.00% 

Table  13.  Experimental  performance  matrix  based  on  PROFICIENT 


Table  13  shows  that  there  is  a  significant  difference  between  the 
PROFICIENT  and  the  “dangerous”  quadrant  of  the  performance  matrix.  This  again  goes 
back  to  the  findings  that  the  PROFICIENT  pilots  were  on  average  less  confident  than 
non-PROFICIENT  pilots.  Both  PROFICIENT  and  non-PROFICIENT  were  off  track 
about  the  same  amount  of  time,  but  the  PROFICIENT  pilots  were  able  to  adjust  their 
confidence  better  than  non-PROFICIENT.  Although  PROFICIENT  pilots  were  in  the 
“dangerous”  category  less  often,  22.61%  compared  to  30.74%,  they  were  also  less 
confident  when  they  were  on  course  compared  to  non-PROFICIENT.  Non- 
PROFICIENT  pilots  had  correct  perception  (NAV  On-track  and  High  CONF,  NAV  Off¬ 
track  and  Low  CONF)  65.8%  of  the  time  while  PROFICIENT  pilots  had  correct 
perception  only  60%  of  the  time.  This  shows  that  PROFICIENT  pilots  were  less 
confident  during  the  routes,  but  enough  to  be  statistically  significant.  This  led  to  the  fact 
that  differences  in  the  performance  matrix  exist. 

E.  LEARNING  EFFECTS  OF  EXPERIMENTS 

1.  Learning  Effect  (Past  Experiment  Participants  vs.  First  Timers) 

This  experiment  had  three  participants  who  had  previously  conducted  a  similar 
experiment.  These  participants  could  have  a  learning  effect  because  they  had  already 
seen  a  similar  OTW  view  and  controlled  a  simulated  aircraft.  These  participants  had  seen 
Sullivan’s  (2010)  route,  so  they  could  have  an  advantage  over  other  participants  during 
the  two  autonavigation  routes  that  were  based  off  Sullivan’s  (2010)  route.  Analysis  was 
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conducted  to  see  if  these  three  participants  skewed  the  data.  Figure  20  shows  the  average 
breakdown  of  CONF,  ERROR  1,  and  ERROR2  based  off  of  PAST_EXP. 


Route  1 


Route  2 
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Figure  20.  Comparative  column  graphs  of  CONF,  ERROR1,  and  ERROR2  based  on 
PAST_EXP  for  each  route.  Error  bars  are  for  95%  confidence  level. 


Figure  20  shows  that  the  95%  confidence  level  bars  overlap  for  each  variable  and 
each  route  except  for  CONF  and  ERROR2  on  route  2.  This  shows  that  there  does  not 
seem  to  be  a  significant  learning  effect  for  participants  who  participated  in  a  prior 
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experiment.  Figure  20  does  show  that  those  who  have  completed  prior  experiments  had 
higher  CONF  and  lower  ERROR  1.  To  see  if  these  differences  are  significant,  Student  t 
tests  were  conducted  and  shown  in  Table  14. 


Route 

1 

2 

3 

4 

CONF 

f  2 

1.292 

2.289 

0.922 

0.902 

p-value 

0.266 

0.0620| 

0.408 

0.402 

ERROR1 

f  2 

-1.476 

-0.486 

-0.153 

-1.519 

p-value 

0.236 

0.652 

0.887 

0.155 

ERROR2 

f  2 

-0.592 

-3.026 

0.135 

-2.092 

p-value 

0.596 

0.0105* 

0.901 

0.0566| 

t  p<-10,  *  p  <  .05,  **  p  <  .01,  ***  p  <  .001 


Table  14.  Student  t  tests  comparing  PAST_EXP  for  CONF,  ERROR1,  and  ERROR2 

on  each  route. 


Table  14  shows  that  there  is  little  statistically  significant  differences  in  the  data 
except  for  ERROR2  for  route  2.  ERROR2  significance  shows  that  the  participants  with 
PAST_EXP  actually  had  more  bias  in  their  navigation.  Otherwise,  PAST_EXP  had  no 
effect  on  hypothesis  analysis.  PAST_EXP  was  analyzed  in  the  performance  matrix  in 
Table  15. 


PAST EXP 

NAV/CONF BIN 

Yes 

No 

Total 

On-track 

73.56% 

63.27% 

65.35% 

High 

66.67% 

53.06% 

55.81% 

Low 

6.90% 

10.20% 

9.53% 

Off-track 

26.44% 

36.73% 

34.65% 

High 

24.14% 

27.70% 

26.98% 

Low 

2.30% 

9.04% 

7.67% 

Total 

100.00% 

100.00% 

100.00% 

Table  15.  Experimental  performance  matrix  based  on  PAST_EXP.  The  Highlighted 
portion  suggests  that  participants  who  conducted  in  a  past  experiment  were 
in  the  “dangerous”  quadrant  a  similar  amount  of  time  as  those  who  did  not. 


As  Table  15  shows,  PAST_EXP  participants  spent  a  higher  percentage  of  time 


On-track,  but  they  had  about  the  same  percentage  in  the  “dangerous”  quadrant  of  the 
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matrix.  This  means  that  when  they  were  Off-track  they  still  believed  they  were  tracking 
on-course.  This  shows  that  PAST_EXP  participants  did  not  skew  the  analysis  of  pilots  in 
the  “dangerous”  quadrant,  meaning  that  PAST_EXP  does  not  have  a  large  effect  on  the 
data.  There  are  some  minor  differences,  but  they  do  not  detract  from  the  conclusions  of 
the  study. 

2.  Between  Scenario  Differences 

This  section  investigates  if  there  is  a  learning  effect  during  the  course  of  the 
experiment.  Paired  t-tests  were  solved  for  CONF,  ERROR1,  and  ERROR2  for  each  of 
the  routes,  and  shown  in  matrix  format  in  Tables  16  through  18. 


CONF  ti4  Q.Q5 


Routes 

1 

2 

3 

4 

1 

— 

2 

2.603 
p-value  0.0219* 

3 

-2.443 
p-value  0.0284* 

-3.674 

p-value 

0.00281** 

4 

1.012 

-1.168 

3.404 

p-value 

0.00428** 

t  p<10,  *  p  <  .05,  **  p  <  .01,  ***  p  <  .001 


Table  16.  T-test  matrix  on  the  learning  effect  of  CONF  and  route 

Table  16  shows  that  there  was  statistically  significantly  different  CONF  between 
some  of  the  routes.  The  cart  shows  that  route  1  and  route  4  had  similar  CONF  data  while 
routes  2  and  3  had  similar  data.  The  difference  in  CONF  is  not  too  surprising,  because 
the  routes  were  set  up  to  have  varying  difficulties.  Route  1  was  supposed  to  be  harder 
than  route  2,  causing  the  CONF  to  be  lower  in  route  2  than  route  1.  With  a  harder  route, 
there  are  more  chances  for  the  participant  to  get  off-track,  thereby  reducing  their  CONF 
level.  Route  3  and  4  were  set  up  to  be  similar,  but  there  is  a  large  difference  in  the  data. 
The  participants’  realization  at  the  end  of  route  3  that  the  autopilot  did  not  follow  the 
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intended  route  may  have  caused  the  difference.  This  caused  the  participant  to  be  less 
confident  in  the  location  of  route  4.  The  data  shows  that  there  was  not  a  trending  effect 
of  increased  or  reduced  confidence  throughout  the  experiment. 


ERROR1  ti4(  o.o5 


Routes 

1 

2 

3 

4 

1 

— 

2 

-1.718 

— 

3 

-2.622 
p-value  0.0201* 

-1.629 

4 

-1.620 

-0.408 

0.199 

— 

t  p<10,  *  p  <  .05,  **  p  <  .01,  ***  p  <  .001 


Table  17.  Correlation  matrix  on  the  learning  effect  of  ERROR  1  and  route 

Table  17  shows  the  surprising  result  of  the  limited  effect  that  the  routes  had  on 
ERROR1.  Participants  had  similar  perception  errors  on  all  of  the  routes. 


ERROR2  ti4(  o.o5 


Routes 

1 

2 

3 

4 

1 

1 

2 

-1.494 

1 

3 

-1.887 
p-value  0.0800f 

-1.341 

1 

4 

-1.544 

-0.722 

0.677 

1 

t  p<.10,  *  p  <  .05,  **  p  <  .01,  ***  p  <  .001 
Table  18.  Correlation  matrix  on  the  learning  effect  of  ERROR2  and  route 


Table  18  shows  limited  statistical  significance  that  the  route  had  on  ERROR2. 
The  participants  had  a  similar  bias  to  the  intended  route  for  every  route. 
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F.  POWER  ANALYSIS 

The  power  of  a  statistical  test  is  the  probability  that  the  test  will  reject  the  null 
hypothesis  when  the  null  hypothesis  is  actually  false,  or  commonly,  the  probability  of  not 
committing  a  Type  II  error.  The  power  analysis  will  conclude  if  the  experimental  sample 
size  was  large  enough  to  give  significant  results. 

Power  analysis  was  conducted  for  the  significant  correlation  coefficients  for 
routes  1  and  4  of  hypothesis  2.  This  analysis  was  for  a  one-sided  less  than  test  with 
sample  size  of  15  and  an  alpha  of  0.05.  p  values  of  -0.65  and  -0.596  correlate  to  a  power 
of  0.858  and  0.776  respectfully.  The  test  will  successfully  reject  the  null  hypothesis 
when  the  null  hypothesis  is  actually  false  85.8%  for  route  1  and  77.6%  for  route  4.  This 
power  is  high  considering  the  small  sample  size  of  the  experiment,  meaning  that  pilot 
bias  toward  the  intended  route  is  likely  (Cohen,  1988). 

G.  POST  TASK  QUESTIONNAIRE  RESULTS 

At  the  completion  of  the  navigation  and  debriefing  portion  of  the  experiment,  the 
participants  were  given  a  Post  Task  Questionnaire  (Appendix).  This  questionnaire  was 
written  to  answer  two  questions.  The  first  was  to  obtain  navigation  techniques  that  the 
more  successful  pilots  used,  while  the  later  was  an  attempt  to  normalize  CONF  levels.  In 
the  attempt  to  normalize  the  CONF  levels,  some  interesting  outcomes  arose.  The  first 
being  that  only  one  participant  felt  that  pilots  were  not  over  reliant  on  navigation 
equipment  like  GPS,  with  six  neutral  responses  and  eight  positive.  The  second  result  of 
the  questionnaire  is  that  12  of  the  15  participants  thought  that  it  was  easy  to  misinterpret 
terrain  during  overland  navigation,  with  the  other  three  responses  being  neutral.  The  last 
questionnaire  output  was  the  most  interesting.  Only  two  participants  (13.3%)  believe  that 
they  are  overconfident  in  their  navigation  skills.  This  is  surprising  considering  the 
percentage  of  time  the  participants  were  in  the  “dangerous”  quadrant  of  flight.  When  the 
participant  was  Off-track,  they  had  a  high  confidence,  or  wrong  perception.  This 
suggests  that  pilots  are  misperceiving  their  overconfidence  during  navigation. 
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H.  SUMMARY 


The  results  of  the  experiment  showed  that  participants  spent  27%  of  the  time  in 
the  “dangerous”  quadrant  of  the  performance  matrix,  second  only  to  On-track  and  High 
CONF.  It  also  showed  that  participants  spent  a  higher  percentage  of  time  in  the 
“dangerous”  quadrant  for  the  autonavigation  routes.  The  next  data  result  was  that  there 
was  no  correlation  between  EEROR1  and  confidence,  yet  there  was  correlation  between 
ERROR2  and  confidence.  This  means  that  participant’s  confidence  did  not  decrease 
when  perception  error  increased,  but  confidence  did  decrease  when  the  perceived 
distance  from  the  route  increased.  Analysis  was  then  conducted  to  see  in  CONF  and 
ERROR1  changed  with  navigation  duration.  CONF  decreased  and  ERROR1  increased 
the  longer  into  the  route  the  participant  was.  Lastly  exploratory  analysis  was  conducted 
to  see  if  there  was  an  experience  or  learning  effect  on  the  experiment.  Data  showed  that 
EXPERTISE  and  PROFICIENT  had  little  effect  on  the  experiment.  Analysis  also 
showed  that  participants  who  completed  similar  experiments  did  not  skew  the  data. 
Lastly  analysis  showed  that  there  was  some  statistically  significant  differences  in  CONF 
between  each  route,  but  these  differences  could  be  explained  with  the  different  route 
complexities,  but  ERROR  1  and  ERROR2  had  little  difference  between  routes. 
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IV.  MODELING  MISPERCEPTION  IN  BAYESIAN 

FRAMEWORK 


This  section  provides  the  insights  gained  by  modeling  pilot  misperception  in  a 
Bayesian  framework.  The  model  classifies  misperception  categories,  and  determines 
when  pilots  are  likely  to  fall  into  these  categories  during  overland  navigation. 

A.  BAYESIAN  MODELING  OF  MISPERCEPTION 

Table  2  is  an  overview  of  the  Bayesian  misperception  modeling  for  overland 
navigation  from  Yang  et  al.  (2011).  Overland  navigation  requires  the  pilot  to  estimate 
their  location  over  the  ground  given  the  outside  terrain  features  presented  to  them.  To 
model  this,  two  variables  are  defined  and  shown  in  Figure  21. 


Figure  21.  Bayesian  misperception  model  variables,  d  is  data,  which  includes  the 
OTW  visual  cues  or  terrain  features.  H  is  the  location  of  the  aircraft  over 

the  ground. 


In  this  model,  pilots  are  required  to  estimate  p(H\d),  where  H  is  the  location  or 
orientation  of  the  aircraft,  and  d  is  the  data  or  visual  cue  that  the  pilot  receives  OTW. 
Then,  p(H\d)  the  probability  of  a  pilot’s  current  location  being  at  H  after  seeing  a  visual 
d  cue  can  be  obtained  by  applying  Bayes’  rule: 


p(H  I  d) 


_ p(d\H)*p(H ) _ 

p(d  I  H )  *  p(H)  +  p( d  l~  H)  *  p{~  H ) 


(1) 


59 


In  equation  (1),  p(H)  is  the  prior  probability,  i.e.,  the  pilot’s  belief  probability  that 
they  are  at  location  //before  seeing  scene  d.  p(d\H)  is  the  conditional  probability  that  the 
pilot  sees  d  from  OTW  when  they  are  at  location  //,  while  p(d\~H)  is  the  conditional 
probability  that  the  pilot  sees  d  OTW  when  they  are  in  a  different  location,  (-//). 
Therefore,  p(~H)  is  equal  to  1-  p(H),  or  the  probability  the  pilot  is  not  at  location  H. 

The  first  misperception  type  is  when  pilots  take  the  map  data  and  try  to  fit  it  into 
what  they  see  OTW.  This  suggests  that  they  are  estimating  p(Hkl)  as  p(d\H), confusing 
inference  with  evidence.  This  misperception  is  prevalent  when  the  pilot  believes  they  are 
on-track,  and  scene  d  is  likely,  or  p(d\H)  ~  1.  The  pilot  will  then  estimates  p(H\d)  is 
p(d\H)  ~  /,  which  incorrectly  overestimates  the  probability.  This  approximation  is 
incorrect  and  leads  the  pilot  to  view  OTW  scenes  in  a  biased  manner  and  to  be 
overconfident. 


A  second  type  of  misperception  occurs  when  pilots  assume  that  the  terrain  that 
they  see  cannot  look  like  terrain  in  a  similar  area.  They  are  assuming  mutually  exclusive 
events  from  evidence,  or  p(d\~H)  =  1  -  p(d\H).  This  assumption  is  incorrect  because 
some  areas  can  have  very  similar  terrain  to  other  areas  causing  a  similar  visual  cue  (e.g., 
a  hill  or  valley),  that  can  be  observed  at  different  locations  p(d\H)  =  p(d\~H).  Pilots 
acting  under  this  misperception  show  a  bias  to  where  they  perceive  they  are  on  the  map 


and  do  not  consider  that  the  location  they  see  out  the  map  could  be  another  spot  on  the 
map,  or  ~H.  This  misperception  is  shown  in  the  equation  (2), 


p{H\d)  = 


_ p(d\H)*p(H) _ 

P(d\H)*p(H)  +  (l-p(d\H))*p(~H) 


(2) 


The  third  misperception  type  is  when  pilots  disregard  visual  cues  that  do  not  fit 


into  their  current  belief.  Pilots  will  also  only  use  visual  cues  that  are  compatible  with 


their  current  perceived  location.  This  means  that  pilots  estimate  p(H\d)  to  equal  the  prior 
probability  p(H).  Not  only  are  pilots  disregarding  visual  cues,  they  are  only  accepting 
visual  cues  that  support  their  perception.  This  misperception  type  can  be  attributed  to 
Inattentional  Blindness  (Simons  &  Chabris,  1999). 
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B.  NAVIGATION  PERFORMANCE  AND  CONFIDENCE  COMPARISON 


In  this  section,  an  illustrative  case  study  is  made  of  two  subjects’  navigation 
performances.  Route  3  was  chosen  for  this  comparison  because  the  autonavigation 
allows  for  the  pause  points  to  be  in  the  exact  same  location,  and  the  participants  have  the 
same  OTW  view.  This  comparison  focuses  on  pause  points  3  through  6,  the  valley  that 
nine  of  the  12  participants  missed  in  Sullivan  (2010).  Participant  3  was  chosen  because 
they  showed  correct  perception  during  the  navigation,  while  Participant  5  had  incorrect 
perception.  Both  of  these  participants  were  helicopter  pilots  with  over  1,000  total  flight 
hours  and  neither  had  participated  in  a  previous  experiment.  Figure  22  shows  Participant 
3  and  5’s  navigation  performance  for  route  3. 


Figure  22.  Route  3  data  for  Participants  3  and  5.  Black  line  is  the  intended  route  of 
flight,  yellow  line  is  the  auto-pilot  flight  path,  red  star  is  perceived  aircraft 
location  with  pause  point  number  and  confidence  label,  and  blue  circles  are 
actual  aircraft  location  with  pause  point  label.  Notice  Participant  3  has 
correct  perception  while  Participant  5  believes  aircraft  followed  intended 

route. 


Figure  22  shows  the  misperception  of  Participant  5  versus  Participant  3. 
Participant  3  was  able  to  realize  that  the  auto-pilot  did  not  follow  the  intended  route,  and 
the  perceived  and  actual  location  was  close.  Conversely,  Participant  5  did  not  realize  that 
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the  aircraft  deviated  from  the  intend  course.  This  caused  Participant  5  to  still  perceive  the 
aircraft’s  location  on  the  intended  route,  as  can  be  seen  with  the  red  stars  close  to  the 
black  line.  Figure  23  and  Table  19  gives  the  breakdown  of  CONF,  ERROR1,  and 
ERROR2  for  Participants  3  and  5. 


CONF 


■  Subject  3 

■  Subject  5 


Pause  Point 


ERROR1 


ERROR2 


■  Subject  3 

■  Subject  5 


Pause  Point 


H  Subject  3 
■  Subject  5 


Pause  Point 


Figure  23.  Comparative  column  graphs  that  show  the  differences  in  CONF,  ERROR1, 
and  ERROR2  for  Participants  3  and  5.  Notice  the  similar  CONF  but  large 
differences  in  ERROR  1  and  ERROR2. 


Route  3  Subjecl 

3 

Roul 

te  3  Subject  5 

Pause 

Point 

CONF 

ERROR 

1  (km) 

ERROR 

2  (km) 

Pause 

Point 

CONF 

ERROR 

1  (km) 

ERROR 

2  (km) 

3 

0.73 

0.3333 

0.1171 

3 

0.84 

0.2669 

0.0606 

4 

0.75 

0.0890 

0.8327 

4 

0.85 

0.5123 

0.3284 

5 

0.9 

0.2764 

0.5304 

5 

0.87 

0.9627 

0.0979 
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6 

0.76 

0.2775 

1.0330 

6 

0.69 

1.6466 

0.0121 

Table  19.  CONF,  ERROR1,  and  ERROR2  for  subjects  3  and  5  during  route  3.  Notice 
the  lower  CONF  and  ERROR1  and  higher  ERROR2  for  subject  3. 

Table  19  shows  the  difference  of  CONF,  ERROR  1  and  ERROR2  between 
Participants  3  and  5.  Participant  3  had  lower  CONF  for  two  of  the  four  pause  points  than 
Participant  5  even  though  they  had  better  perception,  or  lower  ERROR1.  This  outcome  is 
explained  by  our  previous  findings  under  hypothesis  1,  that  there  is  no  correlation 
between  CONF  and  ERROR  1.  Participant  3  also  has  a  much  larger  ERROR2  than 
Participant  5.  A  larger  ERROR2  is  better  for  this  area  of  the  route  because  the  auto-pilot 
is  not  following  the  intended  route.  By  having  a  relatively  low  ERROR2,  Participant  5 
shows  a  bias  towards  the  intended  route.  This  bias  will  be  critical  in  determining  the 
misperception  types  experienced  by  the  participant. 

The  next  step  in  modeling  the  misperception  of  participants  is  to  look  at  the  route 
and  find  key  terrain  features  that  participants  can  use  to  determine  their  location.  Figure 
24  shows  key  terrain  of  route  3. 
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7.  Wrong  Northern 
valley  is  large  and 
has  small  mound  in 
the  middle  of  it 


5.  2"  small  mountain 
in  open  ter  rain  that  Is 
abeam  of  wrong 
q  valley  Limiting 
feature. 


it  9.  More  mountainous 
terrain  and  no  E-W 
valley  North  of  pt  6 


3.  Small  mountain 
after  pt.3  in  open 
area  to  E.  of  pt.4 


4  1°  Western  valley 
after  pt.  3 


8  Wrong  Northern 
valley  continues  to 
another  open  area 


6.  Correct  valley 
leads  to  higher 
terrain  and  peak 
after  pt.S.  Limiting 
feature. 


2.  Large  open  low- 
level  area  after  pt.3 
Limiting  feature. 


-  '-A  ; 

^  f  1  \J 

'  c 

\ 

(. 

'w 

Figure  24.  Key  terrain  features  of  route  3.  These  terrain  features  should  have  been 

used  by  the  participants  to  determine  their  position. 


The  key  terrain  features  in  Figure  24  could  be  used  as  a  guide  to  notify  the 
participant  that  the  autonavigation  is  not  following  the  intended  route  of  flight.  After 
passing  WP  3,  the  participant  should  notice  two  hills  off  the  right  side  of  the  aircraft  (3. 
and  5.  on  Figure  24).  The  participant  needs  to  turn  left  into  the  valley  at  WP  4  after 
passing  the  first  hill,  and  should  have  realized  that  they  have  gone  too  far  if  they  pass  the 
second  hill.  The  valley  at  WP  4  is  also  the  first  valley  to  the  left  after  WP  3.  This  valley 
is  narrow  and  it  has  rising  terrain  after  WP  5.  The  valley  through  which  the  autopilot 
guides  the  participant  is  wide,  and  has  a  small  mound  in  the  middle  (7.  on  Figure  24). 
This  valley  does  not  have  any  rising  terrain  and  continues  to  another  large,  low-level 
open  area.  Key  terrain  9,  on  Figure  24  is  a  mountainous  area  and  not  the  valley  that  one 
would  expect  to  see  at  WP  6. 
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C.  MISPERCEPTION  SIMULATION 

In  this  section  Participant  5’s  route  3  navigation  from  WPs  3  through  7,  or  pause 
points  3  through  6,  are  analyzed  and  modeled.  Starting  at  pause  point  3,  participant  5 
showed  correct  perception,  following  the  autonavigation  of  the  aircraft.  It  is  at  pause 
point  4  where  Participant  5  fell  into  one  of  the  misperception  types.  At  pause  point  4,  the 
aircraft  was  turning  into  a  valley  to  the  North  of  the  intended  route.  At  this  pause  point, 
the  aircraft  was  in  terrain  that  was  similar  to  WP  4,  turning  into  an  East  to  West  valley. 
This  can  be  described  as  p(d\H)  ~  p(d\~H),  where  d  =  valley  and  H  =  WP  4.  In  order  to 
have  correct  perception  at  this  point,  the  participant  needed  to  notice  they  flew  past  the 
opening  to  the  valley  WP  4,  and  not  use  the  small  mountains  off  the  right-hand-side  of 
the  aircraft  to  notice  that  they  were  too  far  north.  It  is  assumed  that  Participant  5  did  not 
notice  these  terrain  features,  and  at  pause  point  4,  they  still  believed  they  were  on-track 
and  had  scenery  that  resembled  WP  4.  Because  Participant  5  was  in  area  that  could 
resemble  WP  4,  they  could  have  overweighed  their  visual  cue  and  not  considered  that 
they  were  in  a  different  valley,  or  ~H.  This  could  have  led  Participant  5  to  misperception 
type  2.  Misperception  type  1  could  have  also  occurred  because  Participant  5  believed 
they  were  on-track  and  a  valley  should  be  seen  at  the  waypoint,  i.e.,  p(d\H)  is  high. 
Figure  25  shows  the  differences  in  prior  and  posterior  probabilities  for  pause  point  4  for 
Participant  5  acting  as  a  Bayesian  agent  and  for  misperception  types  1  and  3.  Figure  25 
assumes  that  p(d\H)  =  p(d\~H)  =  0.9,  which  is  high  because  WP4  and  pause  point  4  are 
similar,  and  p(H)  is  the  CONF  at  pause  point  3  which  is  0.84. 
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1  -I 


Prior  Posterior 


■  Bayesian  Agent 

■  Misperception  1 

■  Misperception  2 


Figure  25.  Column  chart  comparing  Participant  5  during  route  3  at  pause  point  4, 
assuming  a  high  p(cl\H)  =  p(cl\~H)  =  0.9  and p(H)  is  the  CONF  at  pause 
point  3  which  is  0.84.  Notice  that  the  p(H\d),  or  misperception  2,  increases 
from  prior  to  posterior  for  misperception  types. 


Figure  25  shows  that  the  posterior  probability  increases  for  p(H\d)  for  the 
misperception  types  1  and  2  while  the  probability  for  the  Bayesian  agent  stays  the  same. 
This  result  shows  how  Participant  5  could  have  been  acting  under  a  biased  perception 
leading  them  to  miscalculate  their  location,  yet  remain  confident. 

After  the  initial  misperception  of  Participant  5  at  pause  point  4,  they  misperceived 
pause  point  5  and  6.  Both  of  these  pause  point  misperceptions  relate  to  misperception 
type  3.  While  the  aircraft  is  transiting  down  the  wrong  valley,  there  are  visual  cues  that 
would  allow  the  pilot  to  update  their  position  and  correct  their  perception.  The  first 
visual  cue  is  that  the  Northern  valley  is  much  larger  than  the  correct  one  to  the  South. 
This  incorrect  valley  has  a  noticeable  small  mound  in  the  middle  of  it,  and  it  remains  flat 
while  the  correct  valley  is  narrow  with  increasing  terrain  after  waypoint  5.  Although 
Participant  5’s  CONF  did  reduce  from  pause  point  5  to  6,  it  was  still  biased  to  the 
intended  route  of  flight.  It  is  assumed  that  this  reduction  was  mostly  due  to  the  small 
distance  in  perceived  location  between  pause  points  5  and  6.  Misperception  3  means  that 
the  participant  is  only  seeing  terrain  that  helps  their  already  biased  view  that  they  are  on- 


course. 
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D.  SUMMARY 


This  chapter  provided  insights  gained  by  modeling  pilot  perception  and 
misperception  during  overland  navigation  in  a  Bayesian  framework.  Pilots  can  correctly 
update  their  position  in  an  unbiased  manner  using  Bayesian  updating  when  acting  as  a 
true  Bayesian  agent,  and  they  can  fall  into  one  of  the  misperception  types  when  not  acting 
as  a  Bayesian  agent. 
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V.  CONCLUSION  AND  DISCUSSION 


A.  RESEARCH  LIMITATIONS  AND  ASSUMPTIONS 

The  largest  limitations  affecting  this  study  was  the  limited  number  of  available 
participants,  and  potential  bias  of  aviators  who  are  students/  staff  at  the  Naval 
Postgraduate  School,  which  could  narrow  the  range  of  flight  experience  in  the  sample. 
To  partially  overcome  this  limitation,  participants  were  recruited  from  students  and 
faculty  of  the  Naval  Postgraduate  School  with  various  aviation  communities  and  services, 
and  had  a  wide  range  of  experience  levels.  Another  limitation  was  the  use  of  a  simulated 
environment  and  not  actual  aircraft.  However,  this  drawback  was  minimized  by  using 
realistic  outside  and  inside  cockpit  views.  It  has  also  been  shown  that  navigation  in  a 
virtual  environment  correlates  to  real  world  navigation  (Hahn,  2005).  Due  to  safety 
concerns,  this  study  could  not  be  conducted  at  this  altitude  in  a  real  aircraft  over  similar 
terrain. 

Human  confidence  is  complicated.  People  have  different  perspectives  and 
different  baselines  of  confidence.  Because  of  this  variability  between  individuals, 
correlation  of  confidence  is  based  off  a  specific  route  at  a  specific  time.  Self-reported 
confidence  was  assumed  to  be  linear  on  a  0  to  100%  scale. 

Another  limitation  of  the  experiment  was  the  relatively  small  field  of  view  and  no 
depth  perception  of  the  OTW  display.  In  the  real  world,  pilots  have  the  ability  to  scan 
left  and  right,  but  in  this  experiment,  they  were  limited  to  looking  directly  in  front  of  the 
aircraft  with  a  65-degree  field  of  view.  This  made  the  navigational  task  more  challenging 
because  precise  dead  reckoning  requires  the  pilot  to  get  the  intersection  of  at  least  two 
bearing  lines  off  known  terrain  features.  With  only  the  forward  view,  the  baring  lines  are 
at  acute  angles  making  it  hard  to  get  an  exact  pinpoint  of  location.  To  compensate  for 
this,  the  participants  were  informed  that  the  experiment  was  not  seeking  the  exact 
location,  but  rather  the  general  area.  In  addition,  the  challenging  navigational  task 
increased  the  workload  and  stress  that  can  be  compared  to  real  navigation.  Also,  small 
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field  of  views  and  no  depth  perception  are  experienced  in  actual  operating  conditions 
when  using  night  vision  goggles  with  40-degree  field  of  view  limitations. 

Developing  routes  for  the  experiment  is  a  difficult  task  because  the  subjects  have 
varied  navigational  and  flight  experiences.  The  experiment  should  be  set  up  in  a  way  that 
each  participant  has  a  reasonable  expectation  of  both  success  and  failure.  Because  of  this 
fact,  four  routes  including  two  autonavigation  routes  had  to  be  developed  in  order  span 
all  experience  levels.  Another  limitation  of  the  experiment  was  the  required  pause  points. 
The  pause  points  required  stops  in  navigation  that  could  have  an  adverse  effect  on  the 
experiment.  To  reduce  the  effect  of  pausing  the  navigation  task,  participants  were  told 
that  marking  their  perceived  location  and  confidence  should  take  five  seconds  and  not 
more  than  ten.  This  time  limitation  helped  reduce  the  irritation  of  pausing  the  navigation, 
and  also  did  not  allow  the  participant  extra  time  to  study  the  map  and  OTW  display  to 
enhance  their  navigation  performance.  Participants  were  also  polled  after  the  navigation 
task,  and  none  believed  that  the  pause  points  had  a  negative  effect  on  their  navigational 
task.  The  pause  points  also  caused  difficulty  in  participants  who  wanted  to  use  timing  to 
supplement  their  visual  navigation.  The  digital  clock  they  were  provided  on  their 
instrument  cluster  was  constantly  running,  and  it  did  not  stop  when  the  navigation  was 
paused.  Again,  this  forced  the  participants  to  focus  on  their  visual  navigation,  and  not 
dead  reckoning.  Pausing  is  another  aspect  of  this  study  that  could  not  be  performed  in  an 
aircraft. 

The  final  limitation  of  the  experiment  dealt  with  the  map  display.  The  participants 
were  not  allowed  to  use  the  paper  map  that  they  used  for  their  map  study.  This  did  take 
away  from  some  of  the  realism  of  the  simulation.  Real  world  navigation  involves 
multiple  techniques  including  GPS  navigation,  visual  navigation,  and  instrument 
navigation  focusing  on  heading  and  timing.  For  the  purpose  of  this  experiment,  we 
focused  on  the  visual  navigation.  The  map  display  causes  the  participant  to  focus  on  this 
area,  with  minor  use  of  heading  and  timing.  Another  simulation  question  arose  about 
how  to  deal  with  the  orientation  of  the  map.  During  real  world  map  navigation  the 
PNAC  usually  orients  the  map  in  the  direction  of  travel.  This  eases  the  cognitive  tasking 
of  orienting  outside  objects  to  the  helicopter.  Having  to  deal  with  manually  orienting  the 
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map  and  flying,  increased  the  workload  inside  of  the  simulated  cockpit.  For  this 
experiment,  allowing  the  participants  to  change  the  heading  of  the  map  caused  an 
additional  distraction  and  reduced  the  amount  of  normalization  for  the  experiment. 
Having  the  map  automatically  adjust  to  the  helicopter  heading  demonstrated  to  be 
disorienting  to  some  participants  (Sullivan  2010).  Because  they  were  not  the  ones 
moving  the  map,  it  was  hard  for  them  to  regain  their  location  after  moving  from  the  OTW 
to  the  map  display. 

B.  CONCLUSIONS 

This  experiment  showed  that  pilots  have  biased  perception  when  they  are 
executing  low  level  navigation  routes.  The  participants  in  this  study  were  in  the 
“dangerous”  quadrant  of  the  navigation  performance  matrix  (Off-track  and  High  CONF) 
27%  of  the  time,  which  was  second  only  to  On-track  and  High  CONF.  Of  the  time  that 
the  participants  were  Off-track,  they  had  a  corresponding  High  CONF  77.9%  of  the  time. 
This  shows  that  the  participants  overestimated  their  navigation  performance,  yet  in  a  post 
task  questionnaire  only  two  of  the  15  participants  believed  they  were  overconfident  in 
their  navigation  abilities.  This  again  shows  that  there  is  lack  of  correlation  between 
performance  and  confidence.  Just  as  participants  were  biased  towards  overestimates  of 
personal  ability  in  Stone’s  (1993)  research,  the  participants  in  my  experiment 
overestimated  their  navigation  performance  in  the  complex  cognitive  task  of  navigation 
without  their  knowledge.  That  participants  are  overestimating  their  abilities  without  their 
recognition  can  be  the  cause  of  mishaps  and  mission  failures,  just  as  unrecognized  spatial 
disorientation  is  the  most  dangerous  disorientation  and  is  attributed  to  the  most 
disorientation  mishaps. 

The  next  important  result  from  the  study  is  that  there  is  no  correlation  between 
navigation  performance  and  confidence,  yet  there  is  correlation  between  the  distance 
between  perceived  location  and  intended  route  versus  confidence.  This  suggests  that  the 
further  a  pilot  is  away  from  their  perceived  location;  their  corresponding  confidence  does 
not  decrease.  The  correlation  between  perceived  location  and  intended  route  versus 
confidence  suggests  that  the  pilots  would  decrease  their  confidence  the  further  from  their 
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perceived  location  was  from  the  intended  route.  This  shows  that  the  pilots  could  have  a 
bias  towards  the  intended  route  of  flight. 

Data  from  this  experiment  also  showed  that  participant’s  total  flight  hours, 
experience  with  low-level  desert  navigation,  and  whether  the  participant  participated  in 
similar  experiment  had  little  effect  on  the  above  results.  This  result  suggests  that 
navigation  misperception  is  a  symptom  of  traits  ingrained  in  human  nature.  If  this  is  the 
fact,  additional  training  and  technologies  will  have  to  be  developed  in  order  to  override 
this  biased  thinking. 

The  final  conclusion  is  that  the  output  from  this  experiment  could  allow  low-level 
helicopter  navigation  to  be  modeled  in  a  Bayesian  framework  in  order  to  understand  pilot 
misperception.  The  experiment  showed  that  when  a  participant  is  not  acting  as  a 
Bayesian  agent,  they  can  fall  into  one  of  the  three  misperception  types:  1.  Pilots  take  the 
map  data  and  try  to  fit  it  into  what  they  see  OTW.  2.  Pilots  assume  that  the  terrain  that 
they  see  cannot  look  like  terrain  in  a  similar  area.  3.  Pilots  disregard  visual  cues  that  do 
not  fit  into  their  current  belief. 

C.  IMPLEMENTATION 

Direct  implementation  of  the  results  from  this  experiment  to  new  procedures  and 
technologies  is  difficult  because  it  involves  personal  confidence.  The  most  important 
result  from  the  experiment  is  that  there  needs  to  be  training  on  this  subject  to  give  pilots 
the  ability  to  recognize  that  confidence  does  not  correspond  to  correctness  during 
navigation.  A  single  simulator  event,  possibly  conducted  in  conjunction  could  be 
implemented  into  the  Advanced  Helicopter  Flight  Training  at  NAS  Whiting  Field  and 
possibly  to  the  Aviation  Pre-Flight  Indoctrination  (API)  at  NAS  Pensacola,  along  with 
the  corresponding  Army  and  Air  Force  helicopter  training  schools,  based  on  the  finding 
of  this  experiment.  Results  from  this  experiment  could  also  be  added  to  aviation 
physiology  and  safety  center  documents. 

Results  from  this  experiment  also  suggest  that  helicopter  navigation  equipment  is 
important  for  correct  navigation  performance.  Any  improvements  in  navigation 
equipment  technology,  that  would  reduce  the  reliance  on  visual  navigation,  would  relate 
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to  less  mishaps  and  mission  failures.  Current  fleet  navigation  equipment  requires  large 
amount  of  pilot  input.  Reducing  the  pilot  input  requirements  can  allow  the  PNAC  to 
better  execute  other  duties  in  the  cockpit. 

The  last  implementation  of  the  results  from  this  experiment  involves  changing  the 
Go-No-GO  requirements  for  overland  navigation.  Currently  GPS  is  considered  a 
supplemental  navigation  devise,  and  it  is  not  required  to  execute  overland  low-level 
navigation.  This  experiment  shows  that  is  not  uncommon  for  pilots  to  misperceive  their 
location  just  using  visual  navigation,  and  this  misperception  could  be  alleviated  with 
navigational  equipment  like  GPS. 

D.  FUTURE  WORK 

To  enhance  the  results  of  this  experiment  a  larger  sample  size  spanning  different 
experience  and  communities  could  be  used.  The  larger  sample  size  would  allow  for  a 
better  experience  grouping  of  participants  (expert,  intermediate,  and  novice).  Being  able 
to  effectively  group  the  participants  could  provide  insights  into  “overconfident”  or 
“dangerous”  population.  This  could  pinpoint  where  dedicated  time  and  technology  needs 
to  be  spent. 

The  experiment  could  also  be  conducted  under  realistic  operation  environments. 
These  environments  could  be  nighttime,  emergencies,  and  different  weather  conditions. 
Again  this  would  enhance  the  data  for  real  world  operations. 

The  last  future  work  is  to  combine  model  data  to  pilots  as  they  are  flying  in  a 
training  simulator.  This  would  be  a  real-time  interactive  training  system  that  would 
notify  pilots  about  the  dangers  and  occurrence  of  misperception.  This  training  would 
help  reduce  the  amount  of  misperception  during  operational  navigation. 
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APPENDIX 


Pilot  Misperception  During  Overland  Navigation:  Simulation  Exercise  Evaluation 

Welcome  Script 


Date: 


Subject  ID: 


Scheduled  Arrival  Time: 

Actual  Arrival  Time: 

Hello  and  welcome.  Thank  you  for  participating.  We  hope  that  your  participation  will 
ultimately  lead  to  improvements  in  our  understanding  of  pilot  misperception  during 
overland  navigation. 

Today  we’ll  be  asking  you  to  complete  a  four  short  navigation  exercise  using  a  pc -based 
simulation.  Before  and  after  the  navigation,  we’ll  ask  you  to  fill  out  some  short 
questionnaires  related  to  your  background  and  experience.  During  the  navigation  we  will 
be  pausing  the  simulation  and  asking  you  to  pinpoint  where  on  the  map  you  are,  and  at 
what  confidence  (100  very  confident.... 0  very  lost). 

We  hope  to  take  less  than  45  minutes.  We  ask  for  uninterrupted  participation.  During 
the  simulation  exercise  and  when  near  equipment,  please  observe  no  food/drink 
restrictions.  If  you  need  to  use  a  restroom  they  are  located  across  the  breezeway,  through 
the  double  doors  and  to  the  left.  Bottled  water  is  available  in  the  fridge  by  the  door. 

Are  you  ready  to  go  on? 

The  next  step  is  to  make  sure  you  understand  any  risks,  the  voluntary  nature  of 
participation  and  our  efforts  to  protect  your  privacy  with  the  consent  form. 
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Pilot  Misperception  During  Overland  Navigation:  Simulation  Exercise  Evaluation 

Experiment  check  list 


Subject  ID: 

□  E-mail  confirming  date  and  time 

□  Notify  lab  participants  of  data  collection  time 

□  Validate  equipment  hardware  and  software 

o  Screen  brightness  and  contrast  settings 
o  Lab  lighting  conditions 

□  "Experiment  in  Progress"  signs 

□  Bottled  water  in  fridge. 

□  Introductory  Script 

□  Informed  Consent 

□  Background  questionnaire 

□  Map  set  up 

□  Route  brief 

□  Trial  period  instructions 

□  Audio  recording  equipment  (storage  media,  files  naming 
and  backup  scheme) 

□  Navigation  exercise 

□  Save  and  backup  data;  folder  name:  subject  ID  and  date 

□  Post  exercise  questionnaires 

□  Wrap  up  and  thank  you,  contact  information 
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Date: 


Naval  Postgraduate  School 
Consent  to  Participate  in  Research 

Introduction.  You  are  invited  to  participate  in  a  research  study  entitled  “Pilot  Confidence  during 
Helicopter  Overland  Navigation.”  The  purpose  of  the  research  is  to  give  an  understanding  why 
pilots  track  off  course  during  navigation. 

Procedures.  This  study  entails  navigation  through  a  simulated  environment.  You  will  initially  fill 
out  a  questionnaire  on  your  experience  level.  You  will  then  be  given  a  practice  route  to  gain 
familiarity  with  the  system.  Following  the  practice,  you  will  fly  four  navigation  routes  all  preceded 
by  a  map  study.  The  second  two  routes  will  be  flown  on  autopilot,  where  the  autopilot  route  may  or 
may  not  be  on  course.  During  the  navigation  you  will  be  asked  to  pinpoint  your  location  on  a  map, 
and  the  confidence  of  that  location.  You  will  also  be  given  questionnaires  before  and  after  each 
navigation  route.  The  experiment  is  expected  to  take  no  longer  than  45  minutes.  We  are  expecting 
you  to  be  one  out  the  20-30  participants  in  this  study.  Audio  will  be  recorded  so  that  we  can  better 
replay  the  simulation  and  analyze  the  results.  The  recording  will  be  securely  kept  by  the  primary 
investigator,  will  be  kept  confidential,  will  be  reported  in  an  anonymous  fashion,  and  will  be  erased 
after  the  required  holding  time. 

Location.  The  interview/survey/experiment  will  take  place  in  the  MOVES  Institute,  Watkins 
Building,  Rm#212B. 

Cost.  There  is  no  cost  to  participate  in  this  research  study. 

Voluntary  Nature  of  the  Study.  Your  participation  in  this  study  is  strictly  voluntary.  If  you 
choose  to  participate  you  can  change  your  mind  at  any  time  and  withdraw  from  the  study.  You  will 
not  be  penalized  in  any  way  or  lose  any  benefits  to  which  you  would  otherwise  be  entitled  if  you 
choose  not  to  participate  in  this  study  or  to  withdraw. 

Potential  Risks  and  Discomforts.  The  potential  risks  of  participating  in  this  study  are  minimal. 
There  is  a  risk  of  potential  data  mismanagement. 

Anticipated  Benefits.  Anticipated  benefits  from  this  study  are  to  understand  the  reason  for  pilot 
visual  perception  during  overland  navigation;  increasing  the  ability  to  train  future  pilot  and  giving 
recognition  and  better  practices  to  current  pilots.  You  may  not  directly  benefit  from  your 
participation  in  this  research.  The  alternative  to  participating  in  the  research  is  to  not  participate  in 
the  research. 

Compensation  for  Participation.  No  tangible  compensation  will  be  given. 

Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this  study  will  be  kept 
confidential  to  the  full  extent  permitted  by  law.  All  efforts,  within  reason,  will  be  made  to  keep 
your  personal  information  in  your  research  record  confidential  but  total  confidentiality  cannot  be 
guaranteed.  All  references  to  data  collected  will  be  made  anonymous.  Your  name  will  be 
encoded  as  a  participant  number.  Only  principle  investigators  will  have  access  to  this  key  that 
translates  to  an  identification  number  to  your  name. 
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If  you  consent  to  be  identified  by  name  in  this  study,  any  reference  to  or  quote  by  you  will  be 
published  in  the  final  research  finding  only  after  your  review  and  approval.  If  you  do  not  agree, 
then  you  will  be  identified  broadly  by  discipline  and/or  rank,  (for  example,  “fire  chief’). 

I  1 1  consent  to  be  identified  by  name  in  this  research  study. 

Q  1  do  not  consent  to  be  identified  by  name  in  this  research  study. 

Points  of  Contact.  If  you  have  any  questions  or  comments  about  the  research,  or  you  experience 
an  injury  or  have  questions  about  any  discomforts  that  you  experience  while  taking  part  in  this 
study  please  contact  the  Principal  Investigator,  Dr.  Ji  Hyun  Yang,  jvanl@nps.edu.  Questions 
about  your  rights  as  a  research  subject  or  any  other  concerns  may  be  addressed  to  the  Navy 
Postgraduate  School  IRB  Chair,  CAPT  John  Schmidt,  USN,  831-656-3864,  jkschmid@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have  been  given  the 
opportunity  to  ask  questions  and  all  the  questions  have  been  answered  to  my  satisfaction.  I  have 
been  provided  a  copy  of  this  form  for  my  records  and  I  agree  to  participate  in  this  study.  I 
understand  that  by  agreeing  to  participate  in  this  research  and  signing  this  form,  I  do  not  waive 
any  of  my  legal  rights. 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 
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Pilot  Misperception  During  Overland  Navigation:  Simulation  Exercise  Evaluation 

Background  Questionnaire 


We  are  interested  in  learning  about  your  navigation  and  flight  experiences. 

1.  Please  provide  the  following  information: 

Age  Gender 

The  following  questions  ask  about  your  navigation  experiences. 

2.  To  what  extent  have  you  participated  in  activities  other  than  overland  navigation  that 
may  contribute  to  improved  navigation  skills?  (Examples  may  include  sport 
orienteering,  land  navigation  exercises,  boy/girl  scouts  etc.)? 

O  O  O  O  O 

No  Very  Limited  Limited  Somewhat  Significant 

Related  Related  Related  Significant  Related 

Experience  Experience  Experience  Experience  Experience 

3.  At  your  peak  of  currency,  how  would  you  rate  your  navigation  skills  in  a  low-level 
(below  200’  AGL)  overland  environment? 

O  O  O  O  O 

Poor  Fair  Average  Good  Excellent 

4.  If  tasked  today,  how  would  you  rate  your  navigation  skills  in  a  low-level  (below  200’ 
AGL)  overland  environment? 

O  O  O  O  O 

Poor  Fair  Average  Good  Excellent 

5.  How  much  experience  do  you  have  with  low  level  navigation  in  mountainous  desert 
terrain? 

O  O  O  O  O 

None  Very  Little  Somewhat  Considerable  Extensive 

6.  How  much  low  level  navigation  experience  do  you  have  in  the  29  Palms  area? 

O  O  O  O  O 

None  Very  Little  Somewhat  Considerable  Extensive 
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Pilot  Misperception  During  Overland  Navigation:  Simulation  Exercise  Evaluation 

Background  Questionnaire 

The  following  questions  ask  about  your  flight  experiences. 

1 .  Please  provide  the  following  information: 

Total  flight  hours: 

Overland  hours: 

Branch  of  Service: 

Community: 

Years  of  aviation  experience: 

2.  How  many  months  has  it  been  since  your  last  flight? 

3.  How  many  months  has  it  been  since  your  last  overland  navigation  flight? 

4.  If  applicable,  how  many  months  has  it  been  since  your  last  search  and  rescue 
mission? 

5.  Describe  your  operational  flying  experience: 

6.  Have  you  participated  in  either  of  the  prior  navigation  studies? 
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Pilot  Misperception  During  Overland  Navigation:  Simulation 
Exercise  Evaluation 

Post  Task  Questionnaire  D 

Subject  ID: 


Route  1: 

If  you  believed  that  you  did  not  hit  every  waypoint: 
What  was  the  reason? 


Did  you  feel  you  misperceived  some  of  the  terrain  features? 


What  techniques  could  you  have  used  in  order  to  not  miss  waypoints  next 
time? 
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Date: 


Pilot  Misperception  During  Overland  Navigation:  Simulation  Exercise 

Evaluation  SubJect  ID: 

Post  Task  Questionnaire 


Route  2: 

If  you  believed  that  you  did  not  hit  every  waypoint: 
What  was  the  reason? 


Did  you  feel  you  misperceived  some  of  the  terrain  features? 


What  techniques  could  you  have  used  in  order  to  not  miss  waypoints  next 

time? 
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Pilot  Misperception  During  Overland  Navigation:  Simulation  Exercise  Evaluation 

Post  Task  Questionnaire 


Route  3: 

If  you  believed  that  you  misinterpreted  your  position: 
What  was  the  reason? 


Did  you  feel  you  misperceived  some  of  the  terrain  features? 


What  techniques  could  you  have  used  in  order  to  gain  better  confidence  of 
position  next  time? 
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Pilot  Misperception  During  Overland  Navigation:  Simulation  Exercise  Evaluation 

Post  Task  Questionnaire 


Route  4: 

If  you  believed  that  you  misinterpreted  your  position: 
What  was  the  reason? 


Did  you  feel  you  misperceived  some  of  the  terrain  features? 


What  techniques  could  you  have  used  in  order  to  gain  better  confidence  of 
position  next  time? 
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Pilot  Misperception  During  Overland  Navigation:  Simulation  Exercise  Evaluation 

Post-Questionnaire 

1.  What  do  you  think  is  the  most  common  reason  why  pilots  stray  off  course? 


o 

O 

O 

O 

O 

Poor  Map 

Overtasked  in 

Overconfident 

Lack  of 

Other 

Study 

the  Cockpit 

in  Their 

Notable 

Navigation 

Terrain 

Abilities 

Features 

2.  If  you  are 

in  a  situation  where  there  is  a  chance 

you  are  not  on  course,  do  you. 

O 

O 

O 

O 

O 

Continue  with 

Return  to  the 

Stop  at  your 

Circle  the  area 

I  never  get  off 

your  current 

point  where 

current  position 

and  hope  to 

course 

track  and  hope 

you  know  you 

and  try  to  make 

break  out  more 

you  will  find  a 

were  on-track 

sense  of  your 

notable  terrain 

major  terrain 

and  start  over 

surrounding 

features 

feature 

3.  Do  you  feel  that  you  are  over-reliant  on  your  navigation  equipment  like  GPS? 

Never 

Very  Little 

Sometimes 

Considerable 

Almost  Always 

4.  Do  you  feel  that  it  is  easy  to  misperceive  terrain  features  when  you  are  on  low 
level  navigation  routes? 


Never  Very  Little  Sometimes  Considerable  Almost  Always 


5.  Do  you  feel  that  you  are  overconfident  in  your  flight  and  or  navigation 
capabilities? 


Never  Very  Little  Sometimes  Considerable  Almost 

6.  Comments:  Always 
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INITIAL  DISTRIBUTION  LIST 


1.  Defense  Technical  Information  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 
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